00000800:00000200:1.1F:1775772737.632261:0:404327:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000a2ac2043] -> 12345-192.168.201.118@tcp (2) 00000800:00000200:3.1F:1775772737.657054:0:404317:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 1, ksnr_connected 2 ksnr_max_conns 1 00000800:00000200:3.1:1775772737.657089:0:404317:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.201.118@tcp ip 192.168.201.118:988 00000800:00000200:3.1:1775772737.657104:0:404317:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 00000000c0f37ca8 type 2, nob 96 niov 1 nkiov 0 00000800:00000200:1.1:1775772737.668099:0:404317:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 2, ksnr_connected 6 ksnr_max_conns 1 00000800:00000200:0.1F:1775772737.699709:0:404317:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 3, ksnr_connected e ksnr_max_conns 1 00010000:00010000:2.0:1775773233.978893:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.978907:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.978916:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773233.978938:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663517952:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773233.979034:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773233.987490:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663517952:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773233.987517:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987548:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987556:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987569:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987578:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987591:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987604:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987612:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773233.987623:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4e00 x1862032663517952/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773274 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773233.987636:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987647:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3173 (0 0) 00000080:00200000:2.0:1775773233.987654:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773233.987691:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xc65:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773233.987698:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xc65:0x0], updating i_size 0 00000080:00200000:2.0:1775773233.987785:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xc65:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773233.987796:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xc65:0x0](ffff98296faef448) for lock 0x12ae90f9442e8662 00000080:00010000:2.0:1775773233.987806:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xc65:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773233.987809:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8662 from it ffffa414081d3df8 00010000:00010000:2.0:1775773233.987812:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.987821:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773233.987852:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773233.987860:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2329, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773233.987903:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773233.987924:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc65:0x0]: losing layout lock 00010000:00010000:2.0:1775773233.987962:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773233.987970:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773233.987972:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8662 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2a9fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773233.987993:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773233.987999:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4e00 x1862032663518208/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773233.988022:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518208:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773233.988134:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773233.994355:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518208:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773233.994446:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xc65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773233.994455:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xc65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773233.994476:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xc65:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773233.994549:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773233.994565:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773233.994570:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773233.994591:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2658,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773233.994646:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.994661:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773233.994672:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773233.994691:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518336:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773233.994765:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.004422:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518336:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.004448:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004477:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004485:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004497:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004508:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004523:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004548:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004555:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.004566:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5f80 x1862032663518336/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773274 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.004578:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004588:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3388 (0 0) 00000080:00200000:2.0:1775773234.004596:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.004635:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd3c:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773234.004642:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd3c:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.004728:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd3c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.004739:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd3c:0x0](ffff98296faef448) for lock 0x12ae90f9442e8669 00000080:00010000:2.0:1775773234.004749:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd3c:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773234.004753:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8669 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.004755:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.004765:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.004799:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.004807:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.004852:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.004873:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xd3c:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.004914:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.004921:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.004924:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8669 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa01b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.004942:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.004949:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5f80 x1862032663518592/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.004985:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518592:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.005122:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.011731:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518592:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.011817:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xd3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.011828:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xd3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.011851:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xd3c:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773234.011923:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.011940:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2690, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.011945:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2690, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.011986:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2690,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.012043:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.012057:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.012122:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.012145:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518720:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.012212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.020609:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518720:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.020633:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020663:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020670:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020681:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020692:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020704:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020717:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020723:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.020733:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6d80 x1862032663518720/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.020744:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020752:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3404 (0 0) 00000080:00200000:2.0:1775773234.020758:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.020789:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd4c:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773234.020795:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd4c:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.020870:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd4c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.020879:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd4c:0x0](ffff98296faef448) for lock 0x12ae90f9442e8670 00000080:00010000:2.0:1775773234.020889:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd4c:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773234.020892:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8670 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.020895:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.020904:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.020931:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.020939:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2690, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.020988:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.021007:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xd4c:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.021049:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.021055:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.021058:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8670 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa053 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.021123:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.021129:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6d80 x1862032663518976/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.021151:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518976:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.021221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.030932:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663518976:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.031044:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xd4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.031054:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xd4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.031119:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xd4c:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773234.031210:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.031227:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5160, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.031232:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5160, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.031253:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5160,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.031317:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.031330:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.031339:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.031358:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519104:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.031424:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.034859:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519104:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.034887:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034915:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034923:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034935:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034945:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034976:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034990:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.034997:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.035010:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4380 x1862032663519104/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.035022:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.035032:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4589 (0 0) 00000080:00200000:2.0:1775773234.035039:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.035107:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x11ed:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.035115:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x11ed:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.035204:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x11ed:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.035216:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x11ed:0x0](ffff98296fae9988) for lock 0x12ae90f9442e8677 00000080:00010000:2.0:1775773234.035226:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x11ed:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.035229:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8677 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.035232:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.035241:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.035271:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.035279:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5160, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.035340:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.035366:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x11ed:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.035413:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.035420:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.035423:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8677 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11ed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa08b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.035447:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.035454:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4380 x1862032663519360/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.035478:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519360:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.035556:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.045556:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519360:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.045642:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x11ed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.045652:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x11ed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.045674:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x11ed:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.045748:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.045765:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8748, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.045770:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8748, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.045791:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8748,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.045846:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.045859:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.045869:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.045889:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519488:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.045974:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.055277:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519488:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.055315:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055346:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055355:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055367:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055378:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055390:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055403:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055411:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.055421:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6300 x1862032663519488/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.055433:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055444:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6433 (0 0) 00000080:00200000:2.0:1775773234.055451:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.055486:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1921:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.055493:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1921:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.055578:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1921:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.055589:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1921:0x0](ffff98296fae9988) for lock 0x12ae90f9442e867e 00000080:00010000:2.0:1775773234.055599:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1921:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.055603:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e867e from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.055606:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.055615:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.055647:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.055656:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8748, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.055701:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.055726:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1921:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.055768:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.055775:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.055778:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e867e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1921:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa0c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.055800:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.055807:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6300 x1862032663519744/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.055828:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519744:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.055908:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.061399:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519744:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.061484:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1921:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.061494:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1921:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.061516:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1921:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.061612:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.061630:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8723, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.061634:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8723, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.061656:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8723,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.061721:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.061737:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.061747:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.061770:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519872:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.061845:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.072437:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663519872:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.072465:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072496:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072504:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072515:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072533:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072550:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072563:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072569:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.072580:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7480 x1862032663519872/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.072591:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072600:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6370 (0 0) 00000080:00200000:2.0:1775773234.072606:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.072647:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x18e2:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.072654:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x18e2:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.072747:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x18e2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.072757:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x18e2:0x0](ffff98296fae9988) for lock 0x12ae90f9442e8685 00000080:00010000:2.0:1775773234.072767:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x18e2:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.072770:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8685 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.072773:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.072781:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.072814:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.072822:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8723, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.072863:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.072885:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x18e2:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.072924:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.072931:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.072933:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8685 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18e2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa0fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.072952:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.072959:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7480 x1862032663520128/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.072995:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520128:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.073114:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.082841:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520128:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.082940:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x18e2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.082967:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x18e2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.082994:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x18e2:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.083128:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.083146:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2717, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.083151:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2717, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.083178:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2717,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.083453:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.083467:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.083477:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.083497:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520256:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.083569:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.086556:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520256:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.086583:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086618:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086627:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086641:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086655:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086669:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086681:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086688:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.086699:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7100 x1862032663520256/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.086713:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086723:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3417 (0 0) 00000080:00200000:2.0:1775773234.086730:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.086768:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd59:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773234.086775:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd59:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.086868:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd59:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.086880:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd59:0x0](ffff98296faef448) for lock 0x12ae90f9442e868c 00000080:00010000:2.0:1775773234.086893:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd59:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773234.086897:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e868c from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.086900:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.086912:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.086971:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.086980:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2717, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.087032:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.087054:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xd59:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.087130:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.087137:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.087139:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e868c lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa133 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.087158:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.087164:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7100 x1862032663520512/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.087196:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520512:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.087277:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.091740:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663520512:192.168.201.118@tcp:36:rm.0 00010000:00010000:2.0:1775773234.282341:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e86c4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa0a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa2f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.282385:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa0a:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.282446:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e86c4 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa0a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa2f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.282454:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.282456:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e86c4 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa0a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa2f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.282486:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.282497:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5c00 x1862032663523584/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.282521:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523584:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.283616:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.288393:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523584:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.288476:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xa0a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.288486:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xa0a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.288508:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xa0a:0x0](ffff98297d9a4b08) 00000080:00200000:2.0:1775773234.288597:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.288614:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8552, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.288618:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8552, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.288638:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8552,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.288726:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.288735:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.288745:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.288765:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523712:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.288833:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.294425:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523712:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.294452:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294482:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294489:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294499:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294508:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294520:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294541:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294547:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.294559:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518b480 x1862032663523712/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.294572:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294583:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6285 (0 0) 00000080:00200000:2.0:1775773234.294590:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.294629:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x188d:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.294636:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x188d:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.294715:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x188d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.294727:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x188d:0x0](ffff98296fae9988) for lock 0x12ae90f9442e86cb 00000080:00010000:2.0:1775773234.294737:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x188d:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.294740:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e86cb from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.294743:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.294751:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.294779:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.294787:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8552, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.294829:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.294851:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x188d:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.294893:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.294899:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.294901:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e86cb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x188d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa32b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.294925:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.294932:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518b480 x1862032663523968/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.294953:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523968:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.295022:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.301229:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663523968:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.301331:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x188d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.301341:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x188d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.301365:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x188d:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.301444:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.301460:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5362, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.301465:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5362, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.301487:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5362,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.301544:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.301556:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.301567:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.301587:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524096:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.301671:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.311352:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524096:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.311378:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311407:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311415:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311426:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311436:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311449:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311456:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311463:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.311473:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965189c00 x1862032663524096/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.311484:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311494:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4690 (0 0) 00000080:00200000:2.0:1775773234.311500:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.311539:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1252:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.311546:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1252:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.311632:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1252:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.311643:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1252:0x0](ffff98296fae9988) for lock 0x12ae90f9442e86d2 00000080:00010000:2.0:1775773234.311653:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1252:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.311657:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e86d2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.311660:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.311669:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.311702:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.311710:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5362, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.311758:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.311787:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1252:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.311830:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.311836:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.311838:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1252:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa363 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.311864:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.311870:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965189c00 x1862032663524352/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.311891:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524352:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.312039:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.320376:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524352:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773234.320468:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1252:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773234.320479:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1252:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773234.320500:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1252:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.320680:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773234.320700:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4786, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773234.320704:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4786, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773234.320732:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4786,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773234.320798:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.320808:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.320818:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773234.320839:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524480:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773234.320917:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773234.327244:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524480:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773234.327272:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327311:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327320:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327332:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327341:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327356:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327365:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327373:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773234.327384:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518ad80 x1862032663524480/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773275 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773234.327397:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327407:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4452 (0 0) 00000080:00200000:2.0:1775773234.327414:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773234.327446:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1164:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773234.327452:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1164:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.327524:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1164:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.327533:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1164:0x0](ffff98296fae9988) for lock 0x12ae90f9442e86d9 00000080:00010000:2.0:1775773234.327543:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1164:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773234.327547:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e86d9 from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.327551:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.327560:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.327589:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.327596:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4786, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.327637:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.327658:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1164:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.327709:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.327717:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.327720:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e86d9 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aa39b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.327748:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.327755:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518ad80 x1862032663524736/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.327780:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663524736:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.329504:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773234.898926:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1573:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773234.899044:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1573:0x0], updating i_size 0 00000080:00200000:2.0:1775773234.899353:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1573:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773234.899366:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1573:0x0](ffff98296fae80c8) for lock 0x12ae90f9442e87dc 00000080:00010000:2.0:1775773234.899377:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1573:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773234.899381:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e87dc from it ffffa414081d3df8 00010000:00010000:2.0:1775773234.899384:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e87dc lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aabb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773234.899395:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e87dc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aabb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.899427:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773234.899434:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6965, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773234.899477:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e87dc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1573:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aabb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773234.899502:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1573:0x0]: losing layout lock 00010000:00010000:2.0:1775773234.899727:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e87dc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aabb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773234.899739:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773234.899741:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e87dc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1573:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aabb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773234.899766:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773234.899775:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965188380 x1862032663539584/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773234.899797:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663539584:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773234.899941:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00010000:00010000:2.0:1775773235.017872:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.017883:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.017893:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.017904:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.017917:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.017925:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.018010:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.018022:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9ce00 x1862032663542400/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.018035:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.018045:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2860 (0 0) 00000080:00200000:2.0:1775773235.018051:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.018128:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xb2c:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773235.018135:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xb2c:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.018514:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xb2c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.018838:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xb2c:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e8814 00000080:00010000:2.0:1775773235.018852:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xb2c:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773235.018856:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8814 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.018860:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.018870:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.018902:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.018911:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1602, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.019050:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.019119:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb2c:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.019175:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.019182:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.019185:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e8814 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aad73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.019206:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.019214:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965189880 x1862032663542656/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.019236:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663542656:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.019509:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773235.111058:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e883e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b82:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aaec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.111126:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b82:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.111188:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e883e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aaec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.111195:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.111198:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e883e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aaec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.111219:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.111226:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518a300 x1862032663544960/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.111261:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663544960:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.111346:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773235.169779:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169789:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169800:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169810:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169823:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169831:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169838:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.169848:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9fb80 x1862032663546624/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.169860:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.169870:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6254 (0 0) 00000080:00200000:2.0:1775773235.169876:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.169903:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x186e:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773235.169911:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x186e:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.170429:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x186e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.170442:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x186e:0x0](ffff98296faeec08) for lock 0x12ae90f9442e8861 00000080:00010000:2.0:1775773235.170452:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x186e:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773235.170456:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8861 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.170459:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.170467:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.170496:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.170504:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8491, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.170641:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.170666:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x186e:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.170709:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.170716:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.170718:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442e8861 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x186e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aafdb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.170744:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.170752:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518b800 x1862032663546880/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.170774:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663546880:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.170895:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000080:00200000:2.0:1775773235.182395:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x11b1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.182416:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x11b1:0x0](ffff982952590908) for lock 0x12ae90f9442e8868 00000080:00010000:2.0:1775773235.182428:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x11b1:0x0](ffff982952590908) 00000080:00010000:2.0:1775773235.182431:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8868 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.182436:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8868 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x11b1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab013 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.182446:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8868 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x11b1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab013 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.182477:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.182484:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4940, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.182733:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8868 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x11b1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab013 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.182763:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x11b1:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.182808:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8868 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x11b1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab013 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.182814:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.182817:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8868 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x11b1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab013 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.182840:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.182849:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518aa00 x1862032663547264/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.182869:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663547264:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.183373:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.301761:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296f041c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550336:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.301845:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xe55:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.301855:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xe55:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.301878:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xe55:0x0](ffff982952595b88) 00000080:00200000:2.0:1775773235.301974:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.301993:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5888, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.301998:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5888, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.302020:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5888,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.302121:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.302131:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.302141:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.302175:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550464:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.302241:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.307803:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550464:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.307827:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307857:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307878:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307891:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307902:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307915:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307924:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307931:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.307951:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965188e00 x1862032663550464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.307964:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.307975:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5003 (0 0) 00000080:00200000:2.0:1775773235.307981:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.308017:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x138b:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773235.308025:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x138b:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.308657:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x138b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.308677:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x138b:0x0](ffff98296fae9148) for lock 0x12ae90f9442e88a7 00000080:00010000:2.0:1775773235.308688:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x138b:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773235.308692:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88a7 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.308695:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.308704:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.308737:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.308745:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5888, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.308791:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.308815:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x138b:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.308860:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.308867:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.308870:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88a7 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x138b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab20b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.308897:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.308905:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965188e00 x1862032663550720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.308928:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550720:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.309108:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.315692:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550720:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.315771:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x138b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.315780:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x138b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.315803:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x138b:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773235.315876:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.315894:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5510, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.315899:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5510, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.315922:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5510,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.316007:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.316017:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.316028:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.316048:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.316207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.321132:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663550848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.321156:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321198:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: --/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321206:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: --/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321218:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321228:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321240:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321248:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321254:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.321275:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965188a80 x1862032663550848/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.321287:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321297:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4814 (0 0) 00000080:00200000:2.0:1775773235.321304:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.321341:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x12ce:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773235.321349:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x12ce:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.321434:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x12ce:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.321443:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x12ce:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e88ae 00000080:00010000:2.0:1775773235.321453:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x12ce:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773235.321456:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88ae from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.321459:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.321466:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.321493:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.321501:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5510, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.321554:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.321578:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x12ce:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.321622:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 1/0,0 mode: --/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.321628:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.321631:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e88ae lrc: 0/0,0 mode: --/PR res: [0x200000403:0x12ce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.321654:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.321661:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965188a80 x1862032663551104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.321681:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.321753:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.326733:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551104:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.326819:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x12ce:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.326828:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x12ce:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.326849:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x12ce:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773235.326922:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.326936:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3754, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.326952:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3754, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.326974:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3754,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.327029:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.327040:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.327050:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.327109:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.327191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.330652:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.330679:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330708:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330717:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330728:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330739:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330751:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330759:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330767:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.330779:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965189500 x1862032663551232/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.330792:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330804:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3936 (0 0) 00000080:00200000:2.0:1775773235.330810:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.330845:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xf60:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773235.330853:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xf60:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.330957:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xf60:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.330970:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xf60:0x0](ffff98296fae9148) for lock 0x12ae90f9442e88b5 00000080:00010000:2.0:1775773235.330981:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xf60:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773235.330984:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88b5 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.330987:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.330996:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.331028:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.331036:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3754, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.331465:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.331494:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xf60:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.331540:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.331548:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.331550:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e88b5 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xf60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab27b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.331576:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.331584:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965189500 x1862032663551488/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.331606:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.331681:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.336026:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663551488:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.336141:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xf60:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.336151:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xf60:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.336432:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xf60:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773235.415407:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xa1c:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773235.415414:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xa1c:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.415514:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xa1c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.415728:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xa1c:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e88df 00000080:00010000:2.0:1775773235.415741:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xa1c:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773235.415745:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88df from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.415748:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e88df lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab3cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.415757:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e88df lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab3cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.415789:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.415797:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1059, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.415843:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e88df lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa1c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab3cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.415867:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa1c:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.415910:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e88df lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab3cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.415917:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.415919:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e88df lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab3cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.415951:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.415958:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965188000 x1862032663553792/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.415983:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663553792:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.416144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773235.454916:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x99d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.454951:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x99d:0x0](ffff98297d9a6c08) for lock 0x12ae90f9442e88f4 00000080:00010000:2.0:1775773235.454964:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x99d:0x0](ffff98297d9a6c08) 00000080:00010000:2.0:1775773235.454968:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88f4 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.454973:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e88f4 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x99d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab473 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.454985:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e88f4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x99d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab473 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.455023:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.455032:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f805, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.455118:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e88f4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x99d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab473 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.455146:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x99d:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.455196:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e88f4 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x99d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab473 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.455204:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.455207:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442e88f4 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x99d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab473 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.455232:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.455243:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965188700 x1862032663554944/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.455288:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965188700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663554944:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.455376:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.462762:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663554944:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.462854:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x99d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.462874:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x99d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.462898:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x99d:0x0](ffff98297d9a6c08) 00000080:00200000:2.0:1775773235.463294:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.463312:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f301, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.463316:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f301, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.463339:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f301,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.463400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.463412:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.463422:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.463442:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555072:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.463508:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.471806:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555072:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.471834:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471864:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471873:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471884:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471896:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471908:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471916:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471927:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.471947:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965189180 x1862032663555072/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.471960:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.471971:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2159 (0 0) 00000080:00200000:2.0:1775773235.471977:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.472008:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x86f:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.472015:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x86f:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.472151:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x86f:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.472173:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x86f:0x0](ffff98296faeba88) for lock 0x12ae90f9442e88fb 00000080:00010000:2.0:1775773235.472183:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x86f:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.472186:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e88fb from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.472189:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.472197:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.472228:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.472236:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f301, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.472294:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.472324:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x86f:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.472369:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.472376:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.472378:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e88fb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab4ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.472401:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.472409:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965189180 x1862032663555328/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.472433:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555328:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.472512:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.482516:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555328:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.482606:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x86f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.482616:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x86f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.482652:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x86f:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.482740:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.482758:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.482763:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.482789:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1041,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.482849:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.482860:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.482871:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.482891:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555456:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.482960:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.487154:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555456:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.487179:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487208:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487216:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487227:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487238:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487251:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487258:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487283:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.487294:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518a680 x1862032663555456/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.487306:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487324:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2529 (0 0) 00000080:00200000:2.0:1775773235.487331:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.487361:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x9e1:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.487368:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x9e1:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.487449:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x9e1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.487461:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x9e1:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8902 00000080:00010000:2.0:1775773235.487470:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x9e1:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773235.487473:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8902 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.487477:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.487486:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.487515:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.487523:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.487565:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.487591:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x9e1:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.487638:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.487645:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.487648:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8902 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab4e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.487667:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.487673:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518a680 x1862032663555712/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.487692:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.487760:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.496833:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555712:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.496919:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x9e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.496929:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x9e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.496952:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x9e1:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.497028:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.497047:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2002, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.497052:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2002, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.497150:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2002,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.497211:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.497222:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.497233:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.497254:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555840:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.497342:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.502964:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663555840:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.502991:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503025:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503034:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503045:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503054:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503080:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503088:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503099:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.503110:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965189f80 x1862032663555840/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.503121:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503130:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3010 (0 0) 00000080:00200000:2.0:1775773235.503136:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.503166:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xbc2:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.503172:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xbc2:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.503254:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xbc2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.503281:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xbc2:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8909 00000080:00010000:2.0:1775773235.503291:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xbc2:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.503294:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8909 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.503296:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.503304:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.503332:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.503340:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2002, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.503382:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.503410:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xbc2:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.503452:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.503459:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.503462:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8909 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab51b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.503485:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.503492:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982965189f80 x1862032663556096/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.503514:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965189f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556096:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.503595:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.511703:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965189f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556096:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.511803:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xbc2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.511814:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xbc2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.511834:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xbc2:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.511906:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.511922:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7326, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.511927:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7326, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.511950:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7326,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.512002:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.512025:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.512035:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.512054:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556224:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.512170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.518903:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556224:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.518930:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.518962:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.518969:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.518980:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.518990:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.519003:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.519012:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.519023:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.519034:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518bb80 x1862032663556224/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.519045:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.519055:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5672 (0 0) 00000080:00200000:2.0:1775773235.519119:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.519157:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1628:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.519165:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1628:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.519253:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1628:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.519535:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1628:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8910 00000080:00010000:2.0:1775773235.519546:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1628:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.519550:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8910 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.519553:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.519563:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.519593:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.519601:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7326, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.519661:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.519690:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1628:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.519947:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.519955:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.519958:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8910 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1628:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.519977:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.519984:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518bb80 x1862032663556480/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.520005:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556480:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.540731:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556480:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.540814:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1628:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.540823:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1628:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.540846:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1628:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.540917:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.540933:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7072, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.540938:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7072, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.540959:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7072,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.541014:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8917 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.541024:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8917 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.541039:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8917 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.541058:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663556608:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.541153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000080:00200000:2.0:1775773235.638881:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x8ae:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.638916:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x8ae:0x0](ffff98297d9a5348) for lock 0x12ae90f9442e892c 00000080:00010000:2.0:1775773235.639457:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x8ae:0x0](ffff98297d9a5348) 00000080:00010000:2.0:1775773235.639461:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e892c from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.639468:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e892c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x8ae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab633 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.639483:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e892c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x8ae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab633 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.639524:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.639533:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f326, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.639592:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e892c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x8ae:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab633 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.639618:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x8ae:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.639680:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e892c lrc: 1/0,0 mode: --/PR res: [0x200000403:0x8ae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab633 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.639688:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.639691:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442e892c lrc: 0/0,0 mode: --/PR res: [0x200000403:0x8ae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab633 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.639723:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.639738:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296518b100 x1862032663558016/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.639763:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296518b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663558016:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.639948:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.730107:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1f00 going to sleep for 36 seconds 00010000:00010000:2.0:1775773235.738334:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738345:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738356:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738367:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738380:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738388:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738395:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.738407:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ceaa00 x1862032663560448/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.738419:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738430:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2856 (0 0) 00000080:00200000:2.0:1775773235.738436:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.738466:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xb28:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.738473:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xb28:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.738656:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xb28:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.738667:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xb28:0x0](ffff98296faeba88) for lock 0x12ae90f9442e895d 00000080:00010000:2.0:1775773235.738677:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xb28:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.738680:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e895d from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.738684:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.738693:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.738723:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.738730:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1594, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.738776:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.738799:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb28:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.738867:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.738874:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.738877:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e895d lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab7bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.738898:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.738906:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93b80 x1862032663560704/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.738928:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663560704:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.743309:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.748922:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663560704:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.749040:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xb28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.749052:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xb28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.749114:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xb28:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.749203:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.749220:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3113, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.749225:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3113, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.749247:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3113,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.749321:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.749332:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.749343:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.749371:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663560832:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.749449:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.756119:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663560832:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.756143:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756181:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xded:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756190:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756201:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756211:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756223:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756232:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756238:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.756254:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93100 x1862032663560832/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.756278:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756288:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3565 (0 0) 00000080:00200000:2.0:1775773235.756294:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.756324:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xded:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.756329:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xded:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.756403:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xded:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.756413:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xded:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8964 00000080:00010000:2.0:1775773235.756422:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xded:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773235.756425:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8964 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.756428:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.756436:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.756463:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.756470:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3113, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.756507:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.756537:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xded:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.756576:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.756583:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.756586:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8964 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xded:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab7f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.756604:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.756610:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93100 x1862032663561088/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.756629:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561088:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.756696:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.760385:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561088:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.760472:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xded:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.760482:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xded:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.760503:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xded:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.760594:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.760611:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3297, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.760616:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3297, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.760636:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3297,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.760698:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.760708:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.760718:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.760744:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561216:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.760823:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.764581:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561216:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.764604:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764631:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764639:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764648:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764658:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764669:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764676:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764682:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.764697:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93480 x1862032663561216/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.764708:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764718:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3707 (0 0) 00000080:00200000:2.0:1775773235.764724:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.764759:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xe7b:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.764765:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xe7b:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.764856:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xe7b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.764867:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xe7b:0x0](ffff98296faeba88) for lock 0x12ae90f9442e896b 00000080:00010000:2.0:1775773235.764877:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xe7b:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.764880:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e896b from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.764883:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.764891:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.764920:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.764929:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3297, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.764986:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.765009:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xe7b:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.765055:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 1/0,0 mode: --/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.765062:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.765090:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e896b lrc: 0/0,0 mode: --/PR res: [0x200000403:0xe7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab82b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.765111:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.765118:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93480 x1862032663561472/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.765143:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561472:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.765214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.771709:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561472:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.771799:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xe7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.771809:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xe7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.771833:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xe7b:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.771901:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.771918:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7333, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.771922:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7333, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.771955:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7333,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.772010:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.772020:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.772031:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.772057:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561600:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.772454:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.779497:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561600:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.779538:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779575:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779584:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779596:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779607:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779621:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779630:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779637:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.779653:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91c00 x1862032663561600/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.779665:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779676:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5675 (0 0) 00000080:00200000:2.0:1775773235.779683:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.779715:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x162b:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.779723:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x162b:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.779810:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x162b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.779820:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x162b:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8972 00000080:00010000:2.0:1775773235.779830:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x162b:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.779833:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8972 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.779836:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.779845:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.779876:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.779884:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7333, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.779929:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.780546:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x162b:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.780596:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.780604:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.780607:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8972 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x162b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab863 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.780630:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.780640:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91c00 x1862032663561856/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.780662:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561856:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.780747:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.787122:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561856:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.787219:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x162b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.787229:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x162b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.787251:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x162b:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.787338:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.787357:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5274, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.787362:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5274, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.787385:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5274,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.787446:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.787455:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.787465:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.787492:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561984:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.794798:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663561984:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.794824:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794854:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794861:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794872:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794882:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794894:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794902:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794909:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.794925:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92d80 x1862032663561984/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.794938:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.794948:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4646 (0 0) 00000080:00200000:2.0:1775773235.794955:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.794989:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1226:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.794996:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1226:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.795158:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1226:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.795170:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1226:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8979 00000080:00010000:2.0:1775773235.795180:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1226:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773235.795183:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8979 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.795186:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.795195:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.795226:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.795235:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5274, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.795316:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.795341:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1226:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.795385:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.795392:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.795394:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8979 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1226:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ab89b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.795414:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.795421:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92d80 x1862032663562240/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.795444:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663562240:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.795519:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.804665:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663562240:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.804755:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1226:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.804764:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1226:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.804787:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1226:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.804862:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.804878:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f513, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.804883:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f513, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.804904:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f513,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.804960:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.804970:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.804981:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.805005:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663562368:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.805132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.809640:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663562368:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.809665:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809693:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809701:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809717:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809728:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809739:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809746:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809753:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.809769:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90380 x1862032663562368/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.809780:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809789:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2315 (0 0) 00000080:00200000:2.0:1775773235.809796:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.809824:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x90b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773235.809830:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x90b:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.809905:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x90b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.809915:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x90b:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8980 00000080:00010000:2.0:1775773235.809924:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x90b:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773235.809928:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8980 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.809931:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.809940:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.809968:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.809975:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f513, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.810018:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e8980 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x90b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ab8d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.810040:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x90b:0x0]: losing layout lock 00000100:00100000:2.0:1775773235.925785:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1f00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.935578:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965cea680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565312:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.935675:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x13dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.935686:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x13dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.935708:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x13dd:0x0](ffff982952590908) 00000080:00200000:2.0:1775773235.935787:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.935808:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2391, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.935813:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2391, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.935835:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2391,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.935898:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.935908:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.935917:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.935939:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565440:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.936015:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.942027:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565440:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.942057:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942133:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942142:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942167:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942177:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942189:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942196:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942203:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.942215:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91880 x1862032663565440/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.942226:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942235:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3254 (0 0) 00000080:00200000:2.0:1775773235.942240:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.942290:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xcb6:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.942298:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xcb6:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.942391:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xcb6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.942403:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xcb6:0x0](ffff98296faeba88) for lock 0x12ae90f9442e89b8 00000080:00010000:2.0:1775773235.942412:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xcb6:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773235.942416:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e89b8 from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.942419:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.942429:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.942457:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773235.942466:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2391, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773235.942515:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.942553:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xcb6:0x0]: losing layout lock 00010000:00010000:2.0:1775773235.942604:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773235.942611:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773235.942626:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e89b8 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xcb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aba93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773235.942649:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773235.942656:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91880 x1862032663565696/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773235.942678:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565696:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773235.942760:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.948172:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565696:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773235.948263:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xcb6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773235.948274:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xcb6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773235.948294:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xcb6:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773235.948361:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773235.948377:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1762, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773235.948381:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1762, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773235.948400:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1762,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773235.948452:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.948462:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.948471:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773235.948489:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565824:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773235.948939:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773235.957722:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663565824:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773235.957747:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957779:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957787:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957804:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957814:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957825:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957833:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957840:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773235.957850:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92300 x1862032663565824/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773276 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773235.957862:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.957871:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2890 (0 0) 00000080:00200000:2.0:1775773235.957877:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773235.957905:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xb4a:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773235.957911:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xb4a:0x0], updating i_size 0 00000080:00200000:2.0:1775773235.958106:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xb4a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773235.958119:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xb4a:0x0](ffff98296faecb08) for lock 0x12ae90f9442e89bf 00000080:00010000:2.0:1775773235.958130:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xb4a:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773235.958133:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e89bf from it ffffa414081d3df8 00010000:00010000:2.0:1775773235.958136:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773235.958293:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e89bf lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2abacb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773235.958325:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00010000:00010000:2.0:1775773236.076442:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076458:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076470:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076481:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076496:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076504:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076511:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773236.076525:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965cead80 x1862032663568896/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773277 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773236.076838:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.076850:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6729 (0 0) 00000080:00200000:2.0:1775773236.076856:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773236.076890:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1a49:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773236.076896:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1a49:0x0], updating i_size 0 00000080:00200000:2.0:1775773236.076989:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1a49:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773236.077000:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1a49:0x0](ffff98296faeba88) for lock 0x12ae90f9442e89f7 00000080:00010000:2.0:1775773236.077010:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1a49:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773236.077014:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e89f7 from it ffffa414081d3df8 00010000:00010000:2.0:1775773236.077017:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.077026:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773236.077056:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773236.077382:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9341, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773236.077435:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773236.077461:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1a49:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.077506:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.077514:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.077516:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e89f7 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1a49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abc8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.077689:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.077698:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92300 x1862032663569152/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.077773:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663569152:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.077856:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.277517:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f481, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773236.277603:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e8a52 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x8fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2abf63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773236.277628:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x8fb:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.277679:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e8a52 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x8fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abf63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.277687:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.277689:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442e8a52 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x8fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abf63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.277711:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.277720:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93800 x1862032663574144/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.277745:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663574144:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.277829:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773236.287591:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ce8a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663574272:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773236.287618:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287651:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287659:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287670:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287682:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287695:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287702:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287710:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773236.287723:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce8a80 x1862032663574272/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773277 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773236.287735:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287748:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5862 (0 0) 00000080:00200000:2.0:1775773236.287756:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773236.287787:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x16e6:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773236.287794:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x16e6:0x0], updating i_size 0 00000080:00200000:2.0:1775773236.287873:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x16e6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773236.287884:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x16e6:0x0](ffff98296faea1c8) for lock 0x12ae90f9442e8a59 00000080:00010000:2.0:1775773236.287893:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x16e6:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773236.287897:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8a59 from it ffffa414081d3df8 00010000:00010000:2.0:1775773236.287900:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.287909:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773236.287952:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773236.287960:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7606, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773236.288006:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773236.288029:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x16e6:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.288509:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.288518:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.288520:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8a59 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x16e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2abf9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.288636:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.288645:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92a00 x1862032663574528/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.288670:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663574528:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.288794:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.351688:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1af5:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.351769:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e8a83 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1af5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac0eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.351780:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.351783:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442e8a83 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1af5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac0eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.351823:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.351835:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90e00 x1862032663576832/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.351862:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663576832:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.351953:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.419917:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1a7f:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.419980:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e8aad lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1a7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac23b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.419988:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.419991:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e8aad lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1a7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac23b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.420016:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.420025:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90a80 x1862032663579136/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.420046:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663579136:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.420156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.531335:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9758, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773236.531342:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9758, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773236.531364:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9758,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773236.531421:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8ad7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.531431:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8ad7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.531440:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8ad7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773236.531463:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663581184:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773236.531539:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773236.541749:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db19c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.717398:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xcf2:0x0]: losing layout lock 00010000:00010000:2.0:1775773236.717487:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e8b16 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac583 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773236.717496:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773236.717499:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e8b16 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ac583 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773236.717553:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773236.717564:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90000 x1862032663584896/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773236.717589:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663584896:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.717684:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000100:00100000:2.0:1775773236.796182:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663586944:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773236.796283:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773236.834151:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x87e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773236.834161:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x87e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773236.834184:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x87e:0x0](ffff98297d9a3a88) 00000080:00200000:2.0:1775773236.834271:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773236.834295:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8279, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773236.834301:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8279, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773236.834325:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8279,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773236.834380:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8b4e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.834389:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8b4e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773236.834398:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8b4e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773236.834418:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663587712:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773236.834553:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773236.896845:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663588352:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773236.896933:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773237.331449:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.331468:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7674, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.331473:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7674, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.331494:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7674,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.331682:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8c0b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.331693:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8c0b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.331703:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8c0b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.331724:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663598080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773237.331834:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.372300:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db16c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.378396:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965188700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663599232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.378420:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378450:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378458:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378473:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378485:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378497:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378505:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378511:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.378522:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965188700 x1862032663599232/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.378633:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378644:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3792 (0 0) 00000080:00200000:2.0:1775773237.378649:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.378681:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xed0:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773237.378686:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xed0:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.378766:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xed0:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.378776:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xed0:0x0](ffff98296faedb88) for lock 0x12ae90f9442e8c20 00000080:00010000:2.0:1775773237.378786:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xed0:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773237.378790:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8c20 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.378793:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.378800:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.378829:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.378836:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3566, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.378881:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.378981:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xed0:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.379036:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.379044:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.379046:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8c20 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xed0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2acdd3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.379295:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.379303:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92680 x1862032663599488/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.379326:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663599488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.379436:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000080:00200000:2.0:1775773237.411403:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5635, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.411468:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5a00/0x12ae90f9442e8c35 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x130c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ace7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.411491:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x130c:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.411662:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5a00/0x12ae90f9442e8c35 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x130c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ace7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.411671:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.411673:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5a00/0x12ae90f9442e8c35 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x130c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ace7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.411699:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.411708:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91f80 x1862032663600640/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.411733:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663600640:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.411851:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.534597:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663604480:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.534689:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.549834:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa6240 going to sleep for 36 seconds 00000080:00200000:2.0:1775773237.574461:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x12af:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773237.574469:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x12af:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.574626:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x12af:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.574637:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x12af:0x0](ffff98296fae8908) for lock 0x12ae90f9442e8c97 00000080:00010000:2.0:1775773237.574647:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x12af:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773237.574650:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8c97 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.574654:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8c97 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x12af:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad18b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.574663:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8c97 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x12af:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad18b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.574693:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.574701:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5448, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.574745:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8c97 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x12af:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad18b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.574770:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x12af:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.574814:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8c97 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x12af:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad18b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.574821:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.574823:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e8c97 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x12af:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad18b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.574846:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.574854:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91180 x1862032663606016/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.574874:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663606016:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.575039:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00010000:00010000:2.0:1775773237.596897:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14de:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad1fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.597035:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x14de:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.597383:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8ca5 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x14de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad1fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.597392:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.597395:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e8ca5 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x14de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad1fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.597422:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.597433:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90a80 x1862032663606784/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.597457:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663606784:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.597755:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.602046:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663606784:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773237.602442:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x14de:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773237.602455:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x14de:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773237.602485:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x14de:0x0](ffff982952593248) 00000080:00200000:2.0:1775773237.602672:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.602693:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.602698:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.602722:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4784,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.602785:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.602794:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.602804:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.602826:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663606912:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773237.603033:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.606370:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663606912:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.606396:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606428:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606436:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606446:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606456:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606467:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606474:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606481:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.606491:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90e00 x1862032663606912/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.606503:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606513:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4451 (0 0) 00000080:00200000:2.0:1775773237.606519:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.606565:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1163:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773237.606572:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1163:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.606664:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1163:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.606677:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1163:0x0](ffff98296fae8908) for lock 0x12ae90f9442e8cac 00000080:00010000:2.0:1775773237.606686:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1163:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773237.606689:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cac from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.606692:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.606701:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.606730:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.606739:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.606790:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.606818:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1163:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.606863:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.606870:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.606873:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cac lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad233 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.606897:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.606918:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90e00 x1862032663607168/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.606940:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607168:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.607011:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.611473:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607168:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773237.611592:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1163:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773237.611603:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1163:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773237.611628:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1163:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773237.611708:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.611725:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7242, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.611730:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7242, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.611752:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7242,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.611810:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.611819:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.611829:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.611850:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607296:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773237.611935:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.615867:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607296:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.615892:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615939:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615948:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615959:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615971:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615984:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615991:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.615997:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.616008:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92a00 x1862032663607296/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.616020:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.616030:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5680 (0 0) 00000080:00200000:2.0:1775773237.616036:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.616363:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1630:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773237.616371:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1630:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.616456:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1630:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.616469:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1630:0x0](ffff98296faedb88) for lock 0x12ae90f9442e8cb3 00000080:00010000:2.0:1775773237.616480:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1630:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773237.616483:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cb3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.616487:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.616497:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.616552:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.616561:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7242, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.616610:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.616633:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1630:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.616682:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.616690:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.616693:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e8cb3 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1630:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad26b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.616719:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.616726:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92a00 x1862032663607552/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.616748:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607552:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.616824:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.620682:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607552:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773237.620775:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1630:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773237.620785:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1630:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773237.620809:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1630:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773237.620893:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.620925:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5226, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.620930:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5226, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.620955:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5226,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.621021:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.621031:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.621040:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.621061:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607680:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773237.621468:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.630207:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607680:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.630242:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630275:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630283:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630293:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630304:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630316:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630324:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630330:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.630340:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93800 x1862032663607680/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.630352:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630362:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4672 (0 0) 00000080:00200000:2.0:1775773237.630367:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.630400:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1240:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773237.630406:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1240:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.630491:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1240:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.630505:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1240:0x0](ffff98296faea1c8) for lock 0x12ae90f9442e8cba 00000080:00010000:2.0:1775773237.630514:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1240:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773237.630518:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cba from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.630521:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.630722:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.630753:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.630762:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5226, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.630820:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.630844:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1240:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.630887:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.630895:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.630897:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8cba lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1240:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad2a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.630924:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.630930:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93800 x1862032663607936/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.630952:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607936:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.631023:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.637628:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663607936:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773237.637711:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1240:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773237.637722:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1240:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773237.637745:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1240:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773237.637821:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.637837:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1368, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.637842:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1368, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.637862:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1368,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.637931:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.637940:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.637950:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.637970:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663608064:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773237.638039:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.647983:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663608064:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.648012:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648043:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648052:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648062:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648134:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648148:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648156:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648162:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.648175:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92300 x1862032663608064/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.648187:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648198:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2693 (0 0) 00000080:00200000:2.0:1775773237.648205:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.648249:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xa85:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773237.648257:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xa85:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.648342:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xa85:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.648353:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xa85:0x0](ffff98296faea1c8) for lock 0x12ae90f9442e8cc1 00000080:00010000:2.0:1775773237.648362:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xa85:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773237.648366:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cc1 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.648369:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.648378:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.648406:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.648413:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1368, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.648458:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.648481:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xa85:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.648523:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.648529:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.648531:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad2db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.648552:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.648560:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92300 x1862032663608320/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.648582:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663608320:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.648653:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.661603:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663608320:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773237.661693:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xa85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773237.661703:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xa85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773237.661724:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xa85:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773237.661802:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773237.661821:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2805, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773237.661826:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2805, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773237.661850:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2805,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773237.662035:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.662045:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.662054:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773237.687635:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663608448:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773237.687665:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687705:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687715:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687726:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687739:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687752:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687760:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687768:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.687779:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91880 x1862032663608448/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.687792:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.687807:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3461 (0 0) 00000080:00200000:2.0:1775773237.687814:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.687847:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd85:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773237.687854:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd85:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.687963:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd85:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.687975:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd85:0x0](ffff98296faea1c8) for lock 0x12ae90f9442e8cc8 00000080:00010000:2.0:1775773237.687986:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd85:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773237.687990:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cc8 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.687994:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.688003:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e8cc8 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad313 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.688033:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.688041:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2805, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.788217:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788238:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788250:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788261:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788275:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788284:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788290:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773237.788301:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982974b2fb80 x1862032663610752/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773237.788314:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788323:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3625 (0 0) 00000080:00200000:2.0:1775773237.788329:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773237.788359:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xe29:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773237.788365:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xe29:0x0], updating i_size 0 00000080:00200000:2.0:1775773237.788445:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xe29:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773237.788454:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xe29:0x0](ffff98296faedb88) for lock 0x12ae90f9442e8cf2 00000080:00010000:2.0:1775773237.788464:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xe29:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773237.788467:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8cf2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773237.788471:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773237.788479:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.788509:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773237.788516:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3132, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773237.788554:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773237.788574:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xe29:0x0]: losing layout lock 00010000:00010000:2.0:1775773237.788613:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773237.788619:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773237.788622:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8cf2 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xe29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773237.788642:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773237.788648:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91880 x1862032663611008/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773237.788668:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663611008:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773237.788769:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773237.901237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa6180 going to sleep for 36 seconds 00000080:00200000:2.0:1775773238.214992:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x145f:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773238.215001:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x145f:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.215163:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x145f:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.215176:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x145f:0x0](ffff98296fae8908) for lock 0x12ae90f9442e8d7e 00000080:00010000:2.0:1775773238.215187:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x145f:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773238.215190:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8d7e from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.215194:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e8d7e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x145f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ad8c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.215205:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e8d7e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x145f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ad8c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.215246:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.215255:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6313, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.215467:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e8d7e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x145f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ad8c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.215491:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x145f:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.215536:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e8d7e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x145f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad8c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.215544:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.215546:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e8d7e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x145f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ad8c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.215570:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.215591:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90380 x1862032663618688/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.215615:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663618688:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.215705:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773238.364689:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8db6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.364700:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8db6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.364713:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e8db6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.364735:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663621504:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.364818:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.380594:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663621760:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.380695:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00010000:00010000:2.0:1775773238.562691:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562703:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562716:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562730:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562747:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562755:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562763:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.562778:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce9500 x1862032663625728/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.562790:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.562801:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4617 (0 0) 00000080:00200000:2.0:1775773238.562809:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.562840:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1209:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773238.562847:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1209:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.563020:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1209:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.563031:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1209:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8e03 00000080:00010000:2.0:1775773238.563051:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1209:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773238.563055:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e03 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.563058:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.563228:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.563267:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.563277:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5216, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.563327:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.563353:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1209:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.563400:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.563408:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.563410:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e8e03 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adceb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.563433:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.563440:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93480 x1862032663625984/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.563467:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663625984:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.563791:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.568825:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663625984:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.568988:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1209:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.569000:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1209:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.569027:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1209:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773238.569400:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.569418:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6676, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.569422:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6676, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.569446:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6676,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.569501:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.569511:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.569520:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.569962:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626112:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.570221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.574759:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626112:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.574785:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574818:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574827:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574839:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574850:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574863:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574871:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574878:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.574906:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93b80 x1862032663626112/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.574919:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.574930:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5397 (0 0) 00000080:00200000:2.0:1775773238.574938:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.574985:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1515:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.574994:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1515:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.575423:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1515:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.575435:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1515:0x0](ffff98296faecb08) for lock 0x12ae90f9442e8e0a 00000080:00010000:2.0:1775773238.575446:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1515:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773238.575449:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e0a from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.575452:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.575460:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.575488:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.575496:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6676, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.575553:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.575578:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1515:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.575628:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.575636:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.575638:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e0a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1515:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.575660:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.575667:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93b80 x1862032663626368/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.575689:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626368:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.575756:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.579761:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626368:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.579845:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1515:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.579855:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1515:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.579876:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1515:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.580456:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.580473:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f507, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.580478:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f507, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.580499:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f507,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.580856:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.580869:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.580879:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.580993:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626496:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.581358:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.585543:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626496:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.585567:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585599:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585607:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585618:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585630:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585643:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585652:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585659:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.585675:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90700 x1862032663626496/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.585687:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585698:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2262 (0 0) 00000080:00200000:2.0:1775773238.585705:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.585733:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x8d6:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.585740:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x8d6:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.585821:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x8d6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.585833:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x8d6:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8e11 00000080:00010000:2.0:1775773238.585842:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x8d6:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773238.585846:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e11 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.585849:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.585858:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.586189:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.586199:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f507, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.586258:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.586282:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x8d6:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.586331:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.586339:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.586341:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e11 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x8d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.586366:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.586373:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90700 x1862032663626752/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.586397:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626752:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.586511:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.591480:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626752:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.591569:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x8d6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.591579:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x8d6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.591600:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x8d6:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.591675:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.591692:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4680, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.591697:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4680, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.591718:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4680,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.591770:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.591781:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.591791:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.591815:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626880:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.591900:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.596575:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663626880:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.596603:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596636:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596645:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596656:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596667:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596679:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596686:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596694:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.596709:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91500 x1862032663626880/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.596722:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596732:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4349 (0 0) 00000080:00200000:2.0:1775773238.596739:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.596777:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x10fd:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.596785:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x10fd:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.596875:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x10fd:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.596899:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x10fd:0x0](ffff98296faecb08) for lock 0x12ae90f9442e8e18 00000080:00010000:2.0:1775773238.596909:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x10fd:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773238.596913:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e18 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.596916:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.596924:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.596952:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.596959:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4680, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.597004:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.597025:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x10fd:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.597097:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.597105:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.597107:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e18 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2add93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.597130:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.597138:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91500 x1862032663627136/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.597161:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627136:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.597286:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.600789:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627136:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.600878:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x10fd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.600973:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x10fd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.601001:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x10fd:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.601379:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.601397:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8581, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.601401:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8581, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.601425:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8581,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.601733:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.601744:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.601755:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.601782:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627264:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.602002:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.608046:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627264:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.608108:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608142:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608150:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608162:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608173:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608186:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608194:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608201:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.608226:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92d80 x1862032663627264/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.608237:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608248:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6299 (0 0) 00000080:00200000:2.0:1775773238.608254:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.608284:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x189b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.608292:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x189b:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.608382:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x189b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.608393:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x189b:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8e1f 00000080:00010000:2.0:1775773238.608405:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x189b:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773238.608408:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e1f from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.608411:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.608419:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.608447:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.608455:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8581, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.608499:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.608521:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x189b:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.608591:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.608599:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.608601:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e8e1f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x189b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2addcb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.608621:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.608628:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92d80 x1862032663627520/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.608649:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627520:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.608726:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.613641:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627520:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.613730:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x189b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.613740:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x189b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.613763:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x189b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.613839:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.613855:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8402, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.613860:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8402, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.613882:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8402,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.613937:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.613947:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.613957:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.613979:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627648:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.614047:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.617959:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627648:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.617988:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618021:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618031:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618042:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618053:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618097:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618115:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618121:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.618137:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91c00 x1862032663627648/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.618150:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618160:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6210 (0 0) 00000080:00200000:2.0:1775773238.618167:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.618203:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1842:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.618220:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1842:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.618319:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1842:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.618331:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1842:0x0](ffff98296faecb08) for lock 0x12ae90f9442e8e26 00000080:00010000:2.0:1775773238.618342:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1842:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773238.618346:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e26 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.618349:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.618357:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.618386:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.618394:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8402, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.618441:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.618465:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1842:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.618510:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.618519:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.618521:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e26 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1842:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.618542:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.618549:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91c00 x1862032663627904/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.618573:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627904:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.618652:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.627121:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663627904:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.627217:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1842:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.627228:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1842:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.627252:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1842:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.627333:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.627352:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f535, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.627357:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f535, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.627380:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f535,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.627438:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.627447:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.627458:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.627482:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628032:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.627559:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.635470:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628032:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.635497:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635528:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: --/PR res: [0x200000403:0x916:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635536:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: --/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635547:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635558:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635571:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635579:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635586:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.635603:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93100 x1862032663628032/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.635615:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635625:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2326 (0 0) 00000080:00200000:2.0:1775773238.635632:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.635669:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x916:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.635677:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x916:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.635767:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x916:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.635778:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x916:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8e2d 00000080:00010000:2.0:1775773238.635788:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x916:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773238.635791:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e2d from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.635794:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.635802:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.635837:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.635846:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f535, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.635981:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.636006:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x916:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.636050:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.636058:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.636061:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e8e2d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x916:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.636339:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.636346:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93100 x1862032663628288/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.636369:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628288:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.636441:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.640895:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628288:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.640987:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x916:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.640997:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x916:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.641022:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x916:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.641155:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.641179:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5137, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.641184:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5137, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.641219:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5137,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.641302:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.641313:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.641323:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.641349:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628416:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.641427:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.644827:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628416:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.644851:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644879:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644897:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644908:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644918:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644931:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644939:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644947:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.644962:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad4700 x1862032663628416/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.644976:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.644986:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4577 (0 0) 00000080:00200000:2.0:1775773238.644992:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.645023:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x11e1:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.645029:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x11e1:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.645146:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x11e1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.645157:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x11e1:0x0](ffff98296faecb08) for lock 0x12ae90f9442e8e34 00000080:00010000:2.0:1775773238.645167:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x11e1:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773238.645171:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e34 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.645174:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.645183:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.645218:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.645226:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5137, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.645271:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.645293:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x11e1:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.645335:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.645342:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.645344:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e34 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x11e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ade73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.645364:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.645371:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad4700 x1862032663628672/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.645392:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628672:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.645461:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.660473:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628672:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.660585:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x11e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.660598:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x11e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.660621:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x11e1:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.660706:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.660724:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.660729:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.660754:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2784,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.660817:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.660827:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.660838:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.660864:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628800:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.660950:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.667608:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663628800:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.667638:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667669:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667678:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667689:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667701:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667715:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667722:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667729:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.667745:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad5180 x1862032663628800/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.667759:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667771:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3451 (0 0) 00000080:00200000:2.0:1775773238.667777:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.667818:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd7b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.667829:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd7b:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.667944:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd7b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.667956:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd7b:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8e3b 00000080:00010000:2.0:1775773238.667969:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd7b:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773238.667972:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e3b from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.667976:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.667985:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.668024:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.668034:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2784, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.668130:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.668161:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xd7b:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.668219:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.668227:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.668229:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e3b lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adeab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.668251:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.668258:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad5180 x1862032663629056/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.668282:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629056:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.668355:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.674814:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629056:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.674922:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xd7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773238.674933:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xd7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773238.674956:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xd7b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.675026:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773238.675045:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9925, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773238.675050:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9925, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773238.675146:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9925,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773238.675203:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.675223:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.675232:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773238.675260:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629184:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773238.675343:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.682566:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629184:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773238.682594:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682630:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682638:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682649:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682661:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682673:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682681:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682688:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773238.682703:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad6680 x1862032663629184/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773279 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773238.682717:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.682727:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/7021 (0 0) 00000080:00200000:2.0:1775773238.682734:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773238.682770:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1b6d:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773238.682778:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1b6d:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.682869:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1b6d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.683020:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1b6d:0x0](ffff98296faeaa08) for lock 0x12ae90f9442e8e42 00000080:00010000:2.0:1775773238.683033:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1b6d:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773238.683036:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8e42 from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.683040:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.683049:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.683145:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.683154:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9925, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.683202:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.683236:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b6d:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.683279:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.683287:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.683289:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e8e42 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2adee3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.683309:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.683316:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad6680 x1862032663629440/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.683337:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629440:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.683410:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773238.703946:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663629440:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773238.903515:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1547:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773238.903524:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1547:0x0], updating i_size 0 00000080:00200000:2.0:1775773238.903738:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1547:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773238.903752:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1547:0x0](ffff98296faecb08) for lock 0x12ae90f9442e8eab 00000080:00010000:2.0:1775773238.903764:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1547:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773238.903768:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8eab from it ffffa414081d3df8 00010000:00010000:2.0:1775773238.903771:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4e00/0x12ae90f9442e8eab lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1547:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ae22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773238.903781:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4e00/0x12ae90f9442e8eab lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1547:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ae22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.903817:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773238.903826:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6877, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773238.903874:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4e00/0x12ae90f9442e8eab lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1547:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ae22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773238.903964:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1547:0x0]: losing layout lock 00010000:00010000:2.0:1775773238.904011:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4e00/0x12ae90f9442e8eab lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1547:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773238.904019:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773238.904022:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4e00/0x12ae90f9442e8eab lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1547:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773238.904045:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773238.904052:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad5f80 x1862032663635200/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773238.904424:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663635200:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773238.904542:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.221450:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296f041180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663645696:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.221522:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000080:00200000:2.0:1775773239.256435:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x18e6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.256446:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x18e6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.256471:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x18e6:0x0](ffff98297d9a1988) 00000080:00200000:2.0:1775773239.256764:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.256783:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3381, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.256788:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3381, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.256809:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3381,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.256954:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.256970:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.256982:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.257008:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663646848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.257370:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.261554:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663646848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.261591:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261625:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261634:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261646:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261658:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261671:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261686:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261695:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.261707:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad7b80 x1862032663646848/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.261720:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.261732:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3749 (0 0) 00000080:00200000:2.0:1775773239.261738:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.261774:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xea5:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.261781:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xea5:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.261991:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xea5:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.262006:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xea5:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8f84 00000080:00010000:2.0:1775773239.262017:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xea5:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.262020:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8f84 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.262023:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.262032:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.262062:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.262070:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3381, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.262302:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.262325:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xea5:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.262368:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.262374:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.262377:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f84 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xea5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae8f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.262397:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.262404:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad7b80 x1862032663647104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.262427:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.262503:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.266604:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647104:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.266693:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xea5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.266706:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xea5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.266733:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xea5:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.266815:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.266833:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4418, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.266838:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4418, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.266867:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4418,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.266954:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.266970:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.266981:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.267002:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.267110:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.270986:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.271010:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271041:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271050:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271061:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271110:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271123:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271136:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271143:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.271155:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad7100 x1862032663647232/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.271167:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271179:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4268 (0 0) 00000080:00200000:2.0:1775773239.271185:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.271224:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x10ac:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.271231:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x10ac:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.271303:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x10ac:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.271312:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x10ac:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e8f8b 00000080:00010000:2.0:1775773239.271321:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x10ac:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.271324:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8f8b from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.271327:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.271336:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.271361:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.271368:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4418, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.271403:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.271422:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x10ac:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.271458:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 1/0,0 mode: --/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.271465:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.271468:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f8b lrc: 0/0,0 mode: --/PR res: [0x200000403:0x10ac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae92b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.271485:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.271491:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad7100 x1862032663647488/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.271511:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.271582:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.274769:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647488:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.274867:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x10ac:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.274889:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x10ac:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.274911:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x10ac:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.274987:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.275005:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8651, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.275010:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8651, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.275032:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8651,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.275324:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.275340:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.275351:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.275373:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647616:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.275452:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.279486:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647616:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.279512:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279559:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279569:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279581:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279591:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279604:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279618:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279625:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.279636:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad7480 x1862032663647616/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.279648:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279658:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6334 (0 0) 00000080:00200000:2.0:1775773239.279665:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.279695:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x18be:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.279703:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x18be:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.279803:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x18be:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.279815:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x18be:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8f92 00000080:00010000:2.0:1775773239.279825:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x18be:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.279829:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8f92 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.279832:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.279842:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.279885:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.279895:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8651, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.279945:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.279969:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x18be:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.280015:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.280022:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.280025:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e8f92 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae963 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.280047:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.280054:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad7480 x1862032663647872/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.280117:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647872:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.280204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.284860:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663647872:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.284972:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x18be:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.284982:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x18be:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.285004:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x18be:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.285131:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.285149:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5302, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.285154:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5302, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.285177:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5302,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.285246:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.285262:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.285271:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.285295:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663648000:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.285379:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.290742:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663648000:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.290774:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290810:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290818:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290829:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290839:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290851:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290864:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290880:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.290892:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad5c00 x1862032663648000/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.290905:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.290915:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4660 (0 0) 00000080:00200000:2.0:1775773239.290921:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.290955:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1234:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.290962:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1234:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.291054:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1234:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.291104:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1234:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e8f99 00000080:00010000:2.0:1775773239.291114:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1234:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.291118:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8f99 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.291121:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.291130:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.291158:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.291166:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5302, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.291222:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.291245:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1234:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.291289:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.291297:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.291299:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8f99 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1234:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ae99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.291321:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.291328:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad5c00 x1862032663648256/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.291351:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663648256:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.291423:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.303147:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663648256:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.303247:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1234:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.303257:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1234:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.303283:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1234:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.303364:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.303381:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f73, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.303386:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f73, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.303409:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f73,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.303465:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8fa0 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.303481:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8fa0 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.303491:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e8fa0 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.303512:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663648384:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.303589:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000080:00200000:2.0:1775773239.332424:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.332443:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7120, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.332448:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7120, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.332470:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7120,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.332742:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e8fb5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.332754:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e8fb5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.332765:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e8fb5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.332787:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663649536:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.332980:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773239.394517:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4600/0x12ae90f9442e8fca lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aeb23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.394803:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.394807:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4600/0x12ae90f9442e8fca lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aeb23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.395025:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.395045:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad5880 x1862032663650944/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.395218:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663650944:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.395374:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.458001:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652608:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.458136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.466715:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652608:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.466744:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466779:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466789:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466801:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466812:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466825:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466834:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466841:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.466852:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032663652608/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.466865:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.466885:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5894 (0 0) 00000080:00200000:2.0:1775773239.466892:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.466932:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1706:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.466939:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1706:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.467027:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1706:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.467038:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1706:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e8fed 00000080:00010000:2.0:1775773239.467047:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1706:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.467050:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8fed from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.467053:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.467062:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.467359:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.467368:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7671, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.467413:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.467434:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1706:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.467476:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.467484:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.467486:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e8fed lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1706:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aec3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.467508:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.467517:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad6300 x1862032663652864/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.467556:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652864:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.467638:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.471912:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652864:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.472416:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1706:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.472425:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1706:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.472449:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1706:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.472523:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.472555:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f915, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.472561:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f915, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.472584:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f915,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.472647:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.472663:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.472675:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.472697:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652992:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.472895:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.477350:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663652992:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.477379:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477412:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477420:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477430:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477441:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477454:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477468:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477475:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.477486:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad7800 x1862032663652992/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.477497:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.477507:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2516 (0 0) 00000080:00200000:2.0:1775773239.477513:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.477972:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x9d4:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.477982:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x9d4:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.478255:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x9d4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.478269:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x9d4:0x0](ffff98296faeba88) for lock 0x12ae90f9442e8ff4 00000080:00010000:2.0:1775773239.478282:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x9d4:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.478286:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8ff4 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.478290:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.478301:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.478334:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.478343:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f915, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.478389:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.478411:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x9d4:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.478454:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.478460:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.478463:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e8ff4 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x9d4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aec73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.478485:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.478493:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad7800 x1862032663653248/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.478515:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653248:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.478725:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.482415:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653248:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.482548:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x9d4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.482559:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x9d4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.482584:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x9d4:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.482665:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.482683:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6339, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.482689:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6339, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.482712:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6339,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.482776:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.482791:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.482800:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.482822:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653376:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.483036:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.490938:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653376:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.490964:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.490995:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: --/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491004:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: --/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491016:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491027:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491040:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491054:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491062:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.491446:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad6a00 x1862032663653376/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.491461:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491473:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5228 (0 0) 00000080:00200000:2.0:1775773239.491480:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.491519:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x146c:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.491534:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x146c:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.491618:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x146c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.491627:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x146c:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e8ffb 00000080:00010000:2.0:1775773239.491636:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x146c:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.491640:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e8ffb from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.491643:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.491653:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.491681:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.491689:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6339, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.491728:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.491747:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x146c:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.491784:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 1/0,0 mode: --/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.491791:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.491794:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e8ffb lrc: 0/0,0 mode: --/PR res: [0x200000403:0x146c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aecab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.491817:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.491823:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad6a00 x1862032663653632/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.491845:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653632:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.492040:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.495812:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653632:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.495900:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x146c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.495910:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x146c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.495931:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x146c:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.496000:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.496015:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.496020:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.496042:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.496138:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.496154:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.496164:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.496182:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653760:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.496263:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.499768:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663653760:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.499794:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499826:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499835:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499847:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499859:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499882:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499895:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499903:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.499914:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad4e00 x1862032663653760/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.499927:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.499937:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2013 (0 0) 00000080:00200000:2.0:1775773239.499944:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.499973:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x7dd:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.499981:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x7dd:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.500115:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x7dd:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.500130:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x7dd:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9002 00000080:00010000:2.0:1775773239.500141:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x7dd:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.500144:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9002 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.500147:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.500157:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.500188:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.500208:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.500258:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.500281:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x7dd:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.500327:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.500334:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.500336:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9002 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x7dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aece3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.500356:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.500363:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad4e00 x1862032663654016/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.500384:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654016:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.500457:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.504942:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654016:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.505026:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x7dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.505037:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x7dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.505060:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x7dd:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.505461:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.505478:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9739, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.505483:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9739, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.505505:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9739,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.505575:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.505590:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.505599:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.505620:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654144:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.505698:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.508558:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654144:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.508585:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508614:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508623:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508634:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508646:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508659:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508673:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508680:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.508690:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad4a80 x1862032663654144/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.508703:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508713:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6878 (0 0) 00000080:00200000:2.0:1775773239.508720:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.508749:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1ade:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.508756:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1ade:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.508837:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1ade:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.508846:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1ade:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e9009 00000080:00010000:2.0:1775773239.508856:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1ade:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.508860:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9009 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.508863:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.508882:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.508911:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.508920:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9739, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.508967:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.508989:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1ade:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.509029:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.509036:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.509038:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9009 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1ade:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.509058:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.509410:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad4a80 x1862032663654400/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.509435:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654400:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.509507:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.513235:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654400:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.513324:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1ade:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.513334:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1ade:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.513356:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1ade:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.513432:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.513449:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2403, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.513453:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2403, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.513475:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2403,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.513534:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.513548:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.513558:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.513577:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654528:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.513650:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.517744:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654528:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.517770:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517801:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517810:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517821:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517832:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517845:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517859:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517877:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.517888:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad5500 x1862032663654528/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.517902:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.517912:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3260 (0 0) 00000080:00200000:2.0:1775773239.517919:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.517947:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xcbc:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.517954:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xcbc:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.518038:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xcbc:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.518047:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xcbc:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9010 00000080:00010000:2.0:1775773239.518057:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xcbc:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.518061:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9010 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.518117:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.518127:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.518157:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.518166:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2403, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.518221:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.518246:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xcbc:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.518293:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.518301:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.518303:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9010 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xcbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed53 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.518325:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.518332:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad5500 x1862032663654784/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.518358:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654784:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.518438:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.523884:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654784:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.523975:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xcbc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.523985:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xcbc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.524010:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xcbc:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.524138:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.524159:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3506, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.524164:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3506, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.524188:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3506,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.524261:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.524277:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.524289:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.524310:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654912:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.524388:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.527616:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663654912:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.527641:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527672:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527681:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527692:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527702:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527714:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527728:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527736:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.527745:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad4000 x1862032663654912/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.527758:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527767:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3762 (0 0) 00000080:00200000:2.0:1775773239.527774:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.527804:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xeb2:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.527811:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xeb2:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.527896:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xeb2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.527906:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xeb2:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e9017 00000080:00010000:2.0:1775773239.527915:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xeb2:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.527918:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9017 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.527921:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.527930:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.527956:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.527964:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3506, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.528002:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.528020:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xeb2:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.528056:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.528110:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.528113:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e9017 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xeb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aed8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.528133:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.528139:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad4000 x1862032663655168/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.528160:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663655168:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.528232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.532366:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663655168:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.532444:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xeb2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.532453:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xeb2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.532474:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xeb2:0x0](ffff98296faee3c8) 00000100:00100000:2.0:1775773239.621136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773239.625403:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3199, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.625480:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9056 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aef83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.625506:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xe18:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.625726:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9056 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aef83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.625734:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.625737:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9056 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aef83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.625763:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.625774:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad4380 x1862032663658624/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.625798:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663658624:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.626022:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.630302:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663658624:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.630392:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xe18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.630402:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xe18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.630423:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xe18:0x0](ffff982952595348) 00000080:00200000:2.0:1775773239.630503:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.630521:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6947, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.630775:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6947, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.630802:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6947,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.630856:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.630957:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.630969:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.630995:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663658752:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.631309:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.634937:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663658752:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.634964:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.634993:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: --/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635001:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: --/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635011:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635022:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635035:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635042:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635049:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.635107:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad6d80 x1862032663658752/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.635122:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635132:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5482 (0 0) 00000080:00200000:2.0:1775773239.635138:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.635173:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x156a:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.635180:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x156a:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.635269:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x156a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.635279:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x156a:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e905d 00000080:00010000:2.0:1775773239.635290:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x156a:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.635294:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e905d from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.635297:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.635306:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.635335:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.635343:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6947, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.635388:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.635410:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x156a:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.635450:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.635457:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.635459:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e905d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x156a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aefbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.635479:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.635485:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981ad6d80 x1862032663659008/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.635507:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659008:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.635587:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.640005:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659008:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.640146:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x156a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.640156:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x156a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.640178:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x156a:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.640272:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.640290:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1291, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.640295:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1291, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.640315:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1291,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.640394:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.640404:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.640414:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.640439:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659136:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.640521:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.643883:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659136:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.643910:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643941:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643949:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643960:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643970:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643981:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643988:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.643994:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.644009:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68700 x1862032663659136/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.644020:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.644030:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2704 (0 0) 00000080:00200000:2.0:1775773239.644036:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.644110:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xa90:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.644116:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xa90:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.644202:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xa90:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.644214:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xa90:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9064 00000080:00010000:2.0:1775773239.644225:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xa90:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.644229:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9064 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.644233:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.644242:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.644273:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.644282:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1291, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.644324:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.644345:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa90:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.644388:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.644396:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.644398:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e9064 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2aeff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.644420:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.644426:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68700 x1862032663659392/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.644448:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659392:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.644530:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.648613:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659392:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.648700:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xa90:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.648709:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xa90:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.648730:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xa90:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.648807:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.648824:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2769, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.648828:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2769, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.648849:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2769,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.648912:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.648921:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.648930:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.648953:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659520:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.649028:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.652564:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659520:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.652595:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652629:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: --/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652638:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: --/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652650:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652661:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652674:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652681:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652688:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.652705:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69180 x1862032663659520/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.652717:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652726:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3393 (0 0) 00000080:00200000:2.0:1775773239.652733:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.652774:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xd41:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773239.652781:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xd41:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.652877:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xd41:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.652888:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xd41:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e906b 00000080:00010000:2.0:1775773239.652899:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xd41:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773239.652903:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e906b from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.652905:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.652913:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.652942:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.652949:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2769, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.652994:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.653017:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xd41:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.653059:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.653674:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.653678:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e906b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af02b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.653703:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.653712:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69180 x1862032663659776/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.653738:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663659776:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.653883:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000080:00200000:2.0:1775773239.693546:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x18e3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.693560:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x18e3:0x0](ffff98296fae21c8) for lock 0x12ae90f9442e9087 00000080:00010000:2.0:1775773239.693571:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x18e3:0x0](ffff98296fae21c8) 00000080:00010000:2.0:1775773239.693574:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9087 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.693578:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442e9087 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x18e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af10b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.693587:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442e9087 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af10b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.693616:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.693624:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8625, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.693666:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442e9087 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18e3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af10b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.693689:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x18e3:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.693732:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442e9087 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x18e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af10b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.693739:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.693741:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442e9087 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x18e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af10b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.693761:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.693767:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6a680 x1862032663661312/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.693789:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663661312:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.693859:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000080:00200000:2.0:1775773239.715389:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1703:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.715397:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1703:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.715477:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1703:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.715487:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1703:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9095 00000080:00010000:2.0:1775773239.715496:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1703:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.715500:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9095 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.715504:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442e9095 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1703:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af17b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.715512:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442e9095 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1703:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af17b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.715693:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.715702:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7664, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.715750:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442e9095 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1703:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af17b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.715776:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1703:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.715816:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442e9095 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1703:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af17b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.715824:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.715826:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442e9095 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1703:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af17b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.715847:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.715856:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69f80 x1862032663662720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.715966:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663662720:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.716239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.720042:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663662720:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.720308:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1703:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.720318:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1703:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.720341:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1703:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.720419:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.720437:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3390, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.720441:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3390, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.720463:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3390,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.720522:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.720672:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.720684:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.720706:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663662848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.720825:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.724710:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663662848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.724739:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724769:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724778:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724789:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724801:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724814:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724823:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724836:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.724848:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6bb80 x1862032663662848/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.724860:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.724943:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3754 (0 0) 00000080:00200000:2.0:1775773239.724949:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.724980:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xeaa:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773239.724986:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xeaa:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.725185:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xeaa:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.725209:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xeaa:0x0](ffff98296fae9148) for lock 0x12ae90f9442e909c 00000080:00010000:2.0:1775773239.725220:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xeaa:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773239.725223:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e909c from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.725227:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.725237:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.725267:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.725274:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3390, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.725321:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.725350:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xeaa:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.725393:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 1/0,0 mode: --/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.725399:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.725402:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e909c lrc: 0/0,0 mode: --/PR res: [0x200000403:0xeaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af1b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.725422:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.725429:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6bb80 x1862032663663104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.725451:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.725661:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.729142:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663104:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.729231:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xeaa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.729241:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xeaa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.729265:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xeaa:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773239.729337:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.729356:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5206, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.729361:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5206, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.729382:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5206,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.729436:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.729445:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.729455:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.729475:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.729548:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.733626:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.733652:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733683:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733691:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733702:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733713:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733726:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733733:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733746:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.733757:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b100 x1862032663663232/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.733769:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733779:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4662 (0 0) 00000080:00200000:2.0:1775773239.733785:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.733817:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1236:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.733824:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1236:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.733922:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1236:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.733933:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1236:0x0](ffff98296faeba88) for lock 0x12ae90f9442e90a3 00000080:00010000:2.0:1775773239.733943:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1236:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773239.733946:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e90a3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.733950:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.733959:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.733987:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.733995:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5206, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.734037:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.734062:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1236:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.734147:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.734154:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.734157:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90a3 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1236:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af1eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.734176:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.734184:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b100 x1862032663663488/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.734215:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.734285:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.738106:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663488:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.738212:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1236:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.738223:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1236:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.738244:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1236:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.738309:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.738324:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6555, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.738329:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6555, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.738348:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6555,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.738403:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.738413:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.738424:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.738442:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663616:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.738519:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.741696:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663616:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.741722:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741751:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741760:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741770:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741783:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741795:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741803:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741816:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.741827:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b480 x1862032663663616/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.741838:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.741847:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5336 (0 0) 00000080:00200000:2.0:1775773239.741854:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.741901:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x14d8:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773239.741908:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x14d8:0x0], updating i_size 0 00000080:00200000:2.0:1775773239.741992:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x14d8:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773239.742003:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x14d8:0x0](ffff98296fae9148) for lock 0x12ae90f9442e90aa 00000080:00010000:2.0:1775773239.742014:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x14d8:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773239.742017:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e90aa from it ffffa414081d3df8 00010000:00010000:2.0:1775773239.742020:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.742029:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.742057:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773239.742369:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6555, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773239.742418:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773239.742446:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x14d8:0x0]: losing layout lock 00010000:00010000:2.0:1775773239.742488:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 1/0,0 mode: --/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773239.742496:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773239.742498:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e90aa lrc: 0/0,0 mode: --/PR res: [0x200000403:0x14d8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2af223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773239.742519:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773239.742819:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b480 x1862032663663872/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773239.742845:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663872:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773239.742993:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.747564:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663663872:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773239.747642:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x14d8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773239.747652:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x14d8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773239.747673:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x14d8:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773239.747744:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773239.747759:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9309, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773239.747764:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9309, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773239.747784:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9309,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773239.747841:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.747851:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.747874:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773239.747894:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663664000:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.747966:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773239.752286:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663664000:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773239.752317:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752354:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752363:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752374:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752386:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752408:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752421:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773239.752433:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69c00 x1862032663664000/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773280 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773239.752447:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442e90b1 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2af25b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773239.752457:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6713 (0 0) 00000080:00200000:2.0:1775773239.752464:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773239.752502:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1a39:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773239.752510:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1a39:0x0], updating i_size 0 00000100:00100000:2.0:1775773239.959473:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f99880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663671680:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773239.959708:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.060751:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1105:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.060759:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1105:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.060777:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1105:0x0](ffff98296fae0908) 00000080:00200000:2.0:1775773240.060958:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.060972:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9950, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.060976:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9950, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.060996:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9950,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.061042:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9175 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.061050:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9175 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.061058:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9175 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.061267:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663674752:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.061373:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.134394:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1a80 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.141810:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518e300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663677824:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773240.141833:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141934:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: --/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141944:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: --/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141954:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141964:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141976:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141984:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.141992:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773240.142003:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518e300 x1862032663677824/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773281 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773240.142016:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.142026:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4596 (0 0) 00000080:00200000:2.0:1775773240.142032:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773240.142060:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x11f4:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773240.142263:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x11f4:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.142345:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x11f4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.142356:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x11f4:0x0](ffff98296fae9148) for lock 0x12ae90f9442e91ad 00000080:00010000:2.0:1775773240.142366:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x11f4:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773240.142370:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e91ad from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.142373:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.142381:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.142413:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.142422:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5075, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.142466:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.142495:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x11f4:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.142770:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 1/0,0 mode: --/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.142778:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.142781:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e91ad lrc: 0/0,0 mode: --/PR res: [0x200000403:0x11f4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afa3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.142807:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.142815:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68380 x1862032663678080/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.142837:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663678080:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.143045:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.298575:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3926, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.298639:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7600/0x12ae90f9442e920f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xf84:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2afd4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.298660:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xf84:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.298702:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7600/0x12ae90f9442e920f lrc: 1/0,0 mode: --/PR res: [0x240000403:0xf84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afd4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.298709:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.298711:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7600/0x12ae90f9442e920f lrc: 0/0,0 mode: --/PR res: [0x240000403:0xf84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afd4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.298732:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.298739:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69880 x1862032663683456/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.298762:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683456:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.298842:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.303729:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683456:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773240.303820:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xf84:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.303832:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xf84:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.303867:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xf84:0x0](ffff98297d9a42c8) 00000080:00200000:2.0:1775773240.303950:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.303969:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2022, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.303974:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2022, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.303995:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2022,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.304114:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.304124:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.304134:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.304163:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683584:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.304244:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.308751:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683584:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773240.308776:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308806:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308815:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308826:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308836:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308849:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308870:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308877:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773240.308889:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6a300 x1862032663683584/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773281 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773240.308901:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.308910:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3020 (0 0) 00000080:00200000:2.0:1775773240.308916:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773240.308953:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xbcc:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773240.308960:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xbcc:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.309046:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xbcc:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.309057:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xbcc:0x0](ffff98296faeec08) for lock 0x12ae90f9442e9216 00000080:00010000:2.0:1775773240.309104:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xbcc:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773240.309108:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9216 from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.309112:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.309120:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.309166:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.309186:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2022, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.309232:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.309255:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xbcc:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.309294:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.309302:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.309305:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9216 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afd83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.309326:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.309333:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6a300 x1862032663683840/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.309355:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683840:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.309427:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.314013:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683840:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773240.314142:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xbcc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.314160:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xbcc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.314191:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xbcc:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773240.314267:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.314284:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.314289:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.314309:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6223,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.314367:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.314377:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.314388:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.314408:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683968:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.314481:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.318525:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663683968:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773240.318557:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318592:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318606:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318617:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318628:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318642:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318650:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318658:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773240.318670:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b800 x1862032663683968/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773281 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773240.318682:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318693:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5170 (0 0) 00000080:00200000:2.0:1775773240.318699:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773240.318729:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1432:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773240.318736:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1432:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.318826:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1432:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.318837:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1432:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e921d 00000080:00010000:2.0:1775773240.318848:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1432:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773240.318852:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e921d from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.318855:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.318864:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.318892:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.318900:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.318943:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.318967:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1432:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.319007:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.319014:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.319016:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442e921d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2afdbb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.319036:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.319042:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b800 x1862032663684224/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.319062:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663684224:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.319783:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.324486:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663684224:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773240.324701:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1432:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.324712:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1432:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.324734:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1432:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773240.324813:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.324831:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5434, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.324836:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5434, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.324947:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5434,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.325005:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9224 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.325015:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9224 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.325025:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e9224 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.325046:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663684352:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.325331:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.482002:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5664, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.482458:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e926a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x131b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b0023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.482497:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x131b:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.482570:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e926a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x131b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.482582:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.482585:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e926a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x131b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.482616:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.482630:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68e00 x1862032663688448/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.482657:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663688448:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.482830:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.540415:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1186:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773240.540423:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1186:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.540519:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1186:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.540540:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1186:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e927f 00000080:00010000:2.0:1775773240.540551:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1186:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773240.540555:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e927f from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.540558:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e927f lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b00cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.540568:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e927f lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b00cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.540597:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.540604:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4854, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.540650:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e927f lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1186:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b00cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.540680:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1186:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.540730:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e927f lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b00cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.540737:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.540740:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e927f lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1186:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b00cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.540761:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.540769:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68a80 x1862032663689600/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.540792:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663689600:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.540890:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.555974:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1f00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.566527:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x194f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.566540:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x194f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.566565:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x194f:0x0](ffff982952590908) 00000080:00200000:2.0:1775773240.566644:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.566663:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9075, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.566669:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9075, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.566691:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9075,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.566749:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e928d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.566760:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e928d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.566773:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442e928d lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.566811:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663690112:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.567122:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773240.612295:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 4/1,0 mode: --/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612311:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 4/1,0 mode: --/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612323:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612336:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612350:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612358:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612365:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773240.612451:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518ea00 x1862032663690880/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773281 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773240.612466:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612478:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4337 (0 0) 00000080:00200000:2.0:1775773240.612484:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773240.612514:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x10f1:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773240.612521:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x10f1:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.612603:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x10f1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.612614:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x10f1:0x0](ffff98296faeec08) for lock 0x12ae90f9442e929b 00000080:00010000:2.0:1775773240.612624:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x10f1:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773240.612627:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e929b from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.612631:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.612640:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.612672:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.612679:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4657, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.612725:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.612751:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x10f1:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.612797:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.612819:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.612822:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e929b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10f1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b01ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.612936:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.612945:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68000 x1862032663691136/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.612970:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663691136:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.613147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.642871:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1f00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.686507:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x160f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.686523:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x160f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.686551:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x160f:0x0](ffff98297d9a5b88) 00000080:00200000:2.0:1775773240.686640:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.686659:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2537, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.686665:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2537, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.686689:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2537,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.686750:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e92be lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.686761:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e92be lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.686771:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e92be lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.686794:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663692800:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.687454:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.796283:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xb1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.796312:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xb1f:0x0](ffff9829525921c8) 00000080:00200000:2.0:1775773240.796410:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.796432:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5885, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.796436:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5885, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.796461:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5885,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.796518:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e92e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.796542:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e92e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.796552:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e92e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.796571:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663695104:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.796640:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.808285:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1357:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.808297:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1357:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.808321:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1357:0x0](ffff982952597448) 00000080:00200000:2.0:1775773240.808409:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.808426:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8807, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.808431:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8807, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.808453:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8807,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.808510:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e92ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.808521:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e92ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.808550:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e92ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.808572:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663695488:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.808646:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773240.845723:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 4/1,0 mode: --/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845733:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 4/1,0 mode: --/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845744:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845754:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845767:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845774:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845780:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773240.845799:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982975d31f80 x1862032663696256/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773281 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773240.845812:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.845822:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5932 (0 0) 00000080:00200000:2.0:1775773240.845827:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773240.845857:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x172c:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773240.845864:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x172c:0x0], updating i_size 0 00000080:00200000:2.0:1775773240.846038:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x172c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773240.846049:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x172c:0x0](ffff98296fae80c8) for lock 0x12ae90f9442e92fd 00000080:00010000:2.0:1775773240.846057:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x172c:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773240.846061:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e92fd from it ffffa414081d3df8 00010000:00010000:2.0:1775773240.846088:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.846097:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.846127:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773240.846134:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7847, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773240.846188:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773240.846213:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x172c:0x0]: losing layout lock 00010000:00010000:2.0:1775773240.846660:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 1/0,0 mode: --/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.846668:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.846670:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e92fd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x172c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b04bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.846691:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.846698:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69c00 x1862032663696512/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.846718:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663696512:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.846926:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773240.911568:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982945cb8000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663698560:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.911648:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00010000:00010000:2.0:1775773240.915425:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442e9327 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b060b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773240.915434:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773240.915436:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442e9327 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b060b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773240.915459:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773240.915467:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1f80 x1862032663698816/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773240.915490:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663698816:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773240.915561:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000080:00200000:2.0:1775773240.920405:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xc5a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773240.920415:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xc5a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773240.920439:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xc5a:0x0](ffff98297d9a5348) 00000080:00200000:2.0:1775773240.920516:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773240.920662:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8064, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773240.920668:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8064, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773240.920691:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8064,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773240.920749:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e932e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.920760:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e932e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773240.920769:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442e932e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773240.920790:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663698944:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773240.921422:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00010000:00010000:2.0:1775773241.257969:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.257979:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.257989:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258000:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258014:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258026:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258033:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773241.258048:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982975d30700 x1862032663710080/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773282 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773241.258244:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258256:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3586 (0 0) 00000080:00200000:2.0:1775773241.258263:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773241.258298:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xe02:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773241.258305:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xe02:0x0], updating i_size 0 00000080:00200000:2.0:1775773241.258395:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xe02:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.258405:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xe02:0x0](ffff98296fae9988) for lock 0x12ae90f9442e93f9 00000080:00010000:2.0:1775773241.258416:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xe02:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773241.258419:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e93f9 from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.258422:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.258431:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.258466:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.258476:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3155, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.258542:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.258568:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xe02:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.258619:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.258626:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.258628:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e93f9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe02:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0c9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.258651:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.258658:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6aa00 x1862032663710336/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.258685:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663710336:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.258794:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773241.266254:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9400 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.266265:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9400 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.266275:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9400 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.266295:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982975d31880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663710464:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.266367:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000002:00010000:2.0:1775773241.288304:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f438,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.288379:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.288391:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.288402:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.288429:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663710848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.288733:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.292235:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x8e6:0x0](ffff9829525921c8) 00000080:00200000:2.0:1775773241.292243:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x8e6:0x0], updating i_size 0 00000080:00200000:2.0:1775773241.292325:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x8e6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.292335:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x8e6:0x0](ffff9829525921c8) for lock 0x12ae90f9442e9407 00000080:00010000:2.0:1775773241.292345:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x8e6:0x0](ffff9829525921c8) 00000080:00010000:2.0:1775773241.292349:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9407 from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.292352:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x8e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b0d0b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.292361:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x8e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b0d0b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.292393:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.292401:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f438, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.292445:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x8e6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b0d0b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.292466:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x8e6:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.294427:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x8e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0d0b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.294441:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.294444:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9407 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x8e6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0d0b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.294479:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.294490:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69500 x1862032663711104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.294569:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663711104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.294716:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.321406:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1194:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.321418:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1194:0x0](ffff9829525921c8) for lock 0x12ae90f9442e9415 00000080:00010000:2.0:1775773241.321429:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1194:0x0](ffff9829525921c8) 00000080:00010000:2.0:1775773241.321433:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9415 from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.321437:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e9415 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b0d7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.321446:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e9415 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b0d7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.321478:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.321485:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4883, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.321679:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e9415 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1194:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b0d7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.321707:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1194:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.321752:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e9415 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0d7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.321759:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.321761:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442e9415 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1194:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0d7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.321784:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.321791:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69c00 x1862032663711872/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.321816:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663711872:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.322024:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773241.472003:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e9462 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b0fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.472099:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xefe:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.472178:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e9462 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.472186:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.472189:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442e9462 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b0fe3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.472215:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.472225:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68000 x1862032663716096/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.472251:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663716096:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.472342:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.507856:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xa80:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773241.507867:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xa80:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773241.507893:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xa80:0x0](ffff982952590908) 00000080:00200000:2.0:1775773241.507980:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773241.507999:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4734, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773241.508003:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4734, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773241.508026:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4734,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.508127:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e9470 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.508138:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e9470 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.508148:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e9470 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.508186:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663716608:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.508332:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.555435:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3972, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.555505:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442e9477 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xfcd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b108b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.555544:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xfcd:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.555592:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442e9477 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xfcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b108b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.555600:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.555602:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442e9477 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xfcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b108b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.555629:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.555639:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69880 x1862032663717248/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.555663:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663717248:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.555739:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.573356:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xfcd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773241.573369:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xfcd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773241.573612:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xfcd:0x0](ffff98296fae00c8) 00000080:00200000:2.0:1775773241.573698:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773241.573718:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4033, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773241.573723:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4033, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773241.573747:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4033,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.573811:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e947e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.573821:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e947e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.574583:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442e947e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.574616:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663717376:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.574707:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.630756:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98294f4e4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663718784:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773241.630856:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x821:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773241.630867:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x821:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773241.630888:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x821:0x0](ffff98297d9a1148) 00000080:00200000:2.0:1775773241.630966:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773241.630985:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f854, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773241.630990:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f854, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773241.631010:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f854,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.631111:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.631122:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.631132:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.631161:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663718912:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.631239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.640415:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663718912:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773241.640441:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640472:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640481:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640492:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640505:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640517:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640536:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640544:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773241.640556:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6ad80 x1862032663718912/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773282 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773241.640569:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640579:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2486 (0 0) 00000080:00200000:2.0:1775773241.640586:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773241.640616:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x9b6:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773241.640622:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x9b6:0x0], updating i_size 0 00000080:00200000:2.0:1775773241.640708:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x9b6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.640719:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x9b6:0x0](ffff98296faef448) for lock 0x12ae90f9442e949a 00000080:00010000:2.0:1775773241.640740:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x9b6:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773241.640744:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e949a from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.640747:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.640757:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.640789:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.640796:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f854, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.640852:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.640878:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x9b6:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.640922:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.640929:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.640932:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e949a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x9b6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b11a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.640952:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.640959:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6ad80 x1862032663719168/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.640983:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719168:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.641052:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.651902:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719168:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773241.651990:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x9b6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773241.651999:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x9b6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773241.652022:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x9b6:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773241.652925:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773241.652944:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5588, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773241.652949:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5588, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773241.652971:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5588,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.653028:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.653040:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.653061:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.653108:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719296:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.653210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.661596:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719296:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773241.661624:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661657:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661667:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661678:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661690:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661704:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661711:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661718:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773241.661730:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69180 x1862032663719296/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773282 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773241.661743:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661754:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4803 (0 0) 00000080:00200000:2.0:1775773241.661761:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773241.661802:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x12c3:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773241.661809:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x12c3:0x0], updating i_size 0 00000080:00200000:2.0:1775773241.661905:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x12c3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.661917:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x12c3:0x0](ffff98296fae9988) for lock 0x12ae90f9442e94a1 00000080:00010000:2.0:1775773241.661927:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x12c3:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773241.661931:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e94a1 from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.661935:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.661944:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.661975:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.661983:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5588, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.662035:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.662061:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x12c3:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.662142:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.662164:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.662166:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442e94a1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12c3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b11db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.662187:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.662194:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69180 x1862032663719552/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.662216:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719552:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.662288:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.670744:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719552:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773241.670841:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x12c3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773241.670852:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x12c3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773241.670873:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x12c3:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773241.670942:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773241.670959:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8646, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773241.670963:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8646, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773241.670984:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8646,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773241.671036:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.671046:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.671056:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773241.671111:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719680:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773241.671204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773241.676413:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663719680:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773241.676444:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676477:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676485:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676497:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676507:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676520:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676537:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676545:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773241.676558:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6a300 x1862032663719680/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773282 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773241.676572:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676585:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6332 (0 0) 00000080:00200000:2.0:1775773241.676592:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773241.676636:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x18bc:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773241.676644:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x18bc:0x0], updating i_size 0 00000080:00200000:2.0:1775773241.676745:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x18bc:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773241.676755:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x18bc:0x0](ffff98296faef448) for lock 0x12ae90f9442e94a8 00000080:00010000:2.0:1775773241.676767:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x18bc:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773241.676772:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e94a8 from it ffffa414081d3df8 00010000:00010000:2.0:1775773241.676775:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773241.676784:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.676814:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773241.676823:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8646, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.677472:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442e94a8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18bc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b1213 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.677500:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x18bc:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.890099:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442e9503 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb97:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b14eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.890157:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb97:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.890213:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442e9503 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b14eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.890220:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.890222:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442e9503 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b14eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.890243:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.890252:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1c00 x1862032663724928/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.890273:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663724928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.890355:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000080:00200000:2.0:1775773241.908763:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8721, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773241.908851:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442e950a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18e1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b1523 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773241.908877:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x18e1:0x0]: losing layout lock 00010000:00010000:2.0:1775773241.908923:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442e950a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b1523 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773241.908935:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773241.908937:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442e950a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18e1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b1523 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773241.908962:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773241.908971:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3100 x1862032663725312/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773241.908993:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663725312:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773241.909113:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773242.403397:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x800:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773242.403413:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x800:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773242.403441:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x800:0x0](ffff98296fae1988) 00000080:00200000:2.0:1775773242.403525:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773242.403542:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9388, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773242.403547:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9388, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773242.403570:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9388,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773242.403630:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e95ea lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773242.403640:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e95ea lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773242.403651:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442e95ea lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773242.403677:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663737344:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773242.403770:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00010000:00010000:2.0:1775773242.628566:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5c00/0x12ae90f9442e9637 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b1e8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773242.628582:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773242.628586:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5c00/0x12ae90f9442e9637 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b1e8b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773242.628618:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773242.628630:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982945cb8700 x1862032663741824/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773242.628660:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982945cb8700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663741824:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773242.628756:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773243.242093:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773243.316009:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316019:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316206:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316220:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316235:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316244:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316251:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773243.316263:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982975d32680 x1862032663762304/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773284 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773243.316274:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316284:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3173 (0 0) 00000080:00200000:2.0:1775773243.316290:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773243.316322:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xc65:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773243.316329:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xc65:0x0], updating i_size 0 00000080:00200000:2.0:1775773243.316414:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xc65:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773243.316424:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xc65:0x0](ffff98296fae9988) for lock 0x12ae90f9442e97b1 00000080:00010000:2.0:1775773243.316435:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xc65:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773243.316439:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e97b1 from it ffffa414081d3df8 00010000:00010000:2.0:1775773243.316441:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773243.316450:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773243.316481:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773243.316490:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2229, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773243.316727:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773243.316753:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xc65:0x0]: losing layout lock 00010000:00010000:2.0:1775773243.316796:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773243.316896:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773243.316899:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442e97b1 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xc65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b2a5b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773243.316922:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773243.316931:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1880 x1862032663762560/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773243.316953:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663762560:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773243.317298:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00000100:00100000:2.0:1775773243.906326:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663783808:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773243.906411:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773243.934477:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e9947 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x16bf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b370b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773243.934513:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x16bf:0x0]: losing layout lock 00010000:00010000:2.0:1775773243.934575:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e9947 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x16bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b370b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773243.934583:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773243.934586:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7200/0x12ae90f9442e9947 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x16bf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b370b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773243.934609:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773243.934615:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0a80 x1862032663784832/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773243.934637:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663784832:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773243.934706:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000080:00200000:2.0:1775773244.539453:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x907:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773244.539464:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x907:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773244.539486:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x907:0x0](ffff982952593248) 00000080:00200000:2.0:1775773244.539689:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773244.539707:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7461, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773244.539712:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7461, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773244.539735:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7461,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773244.539881:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9acf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773244.539891:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9acf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773244.539900:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9acf lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773244.539921:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663806080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773244.540481:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000080:00200000:2.0:1775773245.178251:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2966, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.178257:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2966, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.178279:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2966,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.178340:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.178350:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.178360:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.178379:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663828992:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.178478:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.181723:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663828992:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.181746:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181783:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181792:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181803:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181813:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181824:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181838:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181846:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.181857:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f1180 x1862032663828992/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.181868:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.181877:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3542 (0 0) 00000080:00200000:2.0:1775773245.181883:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.181913:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xdd6:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773245.181919:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xdd6:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.182299:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xdd6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.182313:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xdd6:0x0](ffff98296fae80c8) for lock 0x12ae90f9442e9c65 00000080:00010000:2.0:1775773245.182323:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xdd6:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773245.182327:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9c65 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.182331:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.182341:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.182378:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.182387:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2966, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.182434:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.182459:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xdd6:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.182511:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.182518:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.182520:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442e9c65 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xdd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b4ffb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.182635:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.182644:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1180 x1862032663829248/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.182669:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663829248:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.182800:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.187243:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663829248:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.187333:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xdd6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.187342:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xdd6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.187366:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xdd6:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773245.187441:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.187457:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2136, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.187462:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2136, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.187484:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2136,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.187678:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.187693:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.187705:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.187725:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663829376:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.187938:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.192309:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663829376:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.192339:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192372:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192380:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192391:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192401:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192414:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192426:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192433:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.192444:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f3b80 x1862032663829376/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.192456:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192466:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3077 (0 0) 00000080:00200000:2.0:1775773245.192472:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.192503:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xc05:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773245.192509:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xc05:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.192786:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xc05:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.192799:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xc05:0x0](ffff98296faeec08) for lock 0x12ae90f9442e9c6c 00000080:00010000:2.0:1775773245.192809:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xc05:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773245.192813:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9c6c from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.192816:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.192824:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.192850:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.192858:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2136, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.192901:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.192924:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc05:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.192966:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.192973:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.192976:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442e9c6c lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5033 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.193147:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.193155:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3b80 x1862032663829632/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.193177:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663829632:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.193307:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00010000:00010000:2.0:1775773245.259439:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e9c96 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.259451:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e9c96 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.259462:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442e9c96 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.259488:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982945cba680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663831680:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.259823:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00010000:00010000:2.0:1775773245.283183:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e9ca4 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.283195:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e9ca4 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.283206:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442e9ca4 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.283230:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663832448:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.283321:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773245.313621:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313632:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313644:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313657:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313673:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313682:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313690:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.313703:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93800 x1862032663833216/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.313715:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.313726:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4355 (0 0) 00000080:00200000:2.0:1775773245.313733:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.313856:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1103:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773245.313865:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1103:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.313965:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1103:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.313978:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1103:0x0](ffff98296fae80c8) for lock 0x12ae90f9442e9cb2 00000080:00010000:2.0:1775773245.314251:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1103:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773245.314256:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9cb2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.314260:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.314270:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.314313:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.314322:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4693, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.314375:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.314403:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1103:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.314739:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.314748:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.314751:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442e9cb2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.314940:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.314954:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2d80 x1862032663833472/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.314981:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663833472:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.315330:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773245.785614:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785624:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785636:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785648:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785662:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785670:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785683:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.785694:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce6d80 x1862032663850112/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.785705:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785716:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/7045 (0 0) 00000080:00200000:2.0:1775773245.785722:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.785750:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1b85:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.785827:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1b85:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.785907:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1b85:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.785917:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1b85:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e9de6 00000080:00010000:2.0:1775773245.785927:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1b85:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773245.785930:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9de6 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.785934:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.785943:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.785970:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.786161:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9972, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.786208:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.786237:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b85:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.786278:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.786284:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.786287:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442e9de6 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5c03 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.786305:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.786311:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2a00 x1862032663850368/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.786332:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850368:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.786443:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.793331:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850368:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.793418:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1b85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.793429:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1b85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.793453:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1b85:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.793622:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.793644:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3517, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.793649:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3517, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.793669:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3517,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.793727:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9ded lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.793736:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9ded lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.793747:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442e9ded lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.793847:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850496:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.793966:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.805236:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ce4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850752:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.805330:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xeb7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.805340:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xeb7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.805362:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xeb7:0x0](ffff98297d9a1988) 00000080:00200000:2.0:1775773245.805439:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.805457:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f935, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.805462:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f935, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.805486:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f935,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.805561:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.805571:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.805581:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.805602:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850880:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.805685:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.809654:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663850880:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.809686:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809718:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809726:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809738:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809749:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809774:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809782:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809787:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.809799:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f2680 x1862032663850880/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.809811:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.809825:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2526 (0 0) 00000080:00200000:2.0:1775773245.809831:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.809871:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x9de:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.809879:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x9de:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.809971:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x9de:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.809996:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x9de:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e9df4 00000080:00010000:2.0:1775773245.810007:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x9de:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773245.810011:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9df4 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.810015:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.810024:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.810059:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.810102:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f935, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.810148:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.810172:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x9de:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.810214:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.810221:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.810224:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9df4 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x9de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5c73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.810249:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.810256:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2680 x1862032663851136/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.810279:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851136:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.810358:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.814738:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851136:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.814845:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x9de:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.814855:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x9de:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.814877:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x9de:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.814950:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.814968:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6909, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.814972:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6909, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.815003:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6909,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.815056:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.815096:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.815106:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.815126:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851264:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.815196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.819327:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851264:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.819352:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819386:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819394:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819407:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819416:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819428:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819436:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819443:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.819455:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f0000 x1862032663851264/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.819467:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819477:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5463 (0 0) 00000080:00200000:2.0:1775773245.819484:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.819519:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1557:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773245.819526:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1557:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.819610:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1557:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.819622:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1557:0x0](ffff98296fae9148) for lock 0x12ae90f9442e9dfb 00000080:00010000:2.0:1775773245.819633:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1557:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773245.819636:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9dfb from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.819639:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.819647:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.819674:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.819682:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6909, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.819725:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.819747:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1557:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.819805:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.819811:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.819814:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9dfb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1557:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5cab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.819839:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.819845:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0000 x1862032663851520/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.819866:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851520:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.819939:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.824484:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851520:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.824565:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1557:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.824575:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1557:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.824597:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1557:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773245.824673:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.824691:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3221, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.824696:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3221, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.824722:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3221,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.824793:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.824804:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.824814:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.824835:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851648:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.824906:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.828125:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851648:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.828150:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828181:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828189:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828200:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828211:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828225:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828232:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828239:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.828250:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f1c00 x1862032663851648/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.828262:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828273:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3619 (0 0) 00000080:00200000:2.0:1775773245.828278:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.828308:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xe23:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.828315:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xe23:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.828396:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xe23:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.828409:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xe23:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e9e02 00000080:00010000:2.0:1775773245.828419:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xe23:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773245.828423:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9e02 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.828426:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.828435:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.828461:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.828469:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3221, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.828514:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.828537:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xe23:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.828577:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.828585:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.828588:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442e9e02 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5ce3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.828613:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.828620:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1c00 x1862032663851904/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.828642:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851904:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.828715:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.831788:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663851904:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773245.831873:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xe23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773245.831882:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xe23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773245.831903:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xe23:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.831986:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773245.832004:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7152, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773245.832009:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7152, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773245.832030:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7152,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773245.832259:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.832270:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.832280:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773245.832298:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663852032:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773245.832370:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773245.835256:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663852032:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773245.835280:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835310:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835318:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835329:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835339:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835351:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835360:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835367:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.835378:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f0e00 x1862032663852032/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.835391:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.835404:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5635 (0 0) 00000080:00200000:2.0:1775773245.835411:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.835449:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1603:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773245.835457:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1603:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.835746:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1603:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.835945:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1603:0x0](ffff98296fae9148) for lock 0x12ae90f9442e9e09 00000080:00010000:2.0:1775773245.835957:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1603:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773245.835960:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9e09 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.835964:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.836056:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442e9e09 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1603:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5d1b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.836187:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00010000:00010000:2.0:1775773245.985046:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985055:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985267:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985281:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985299:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985307:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985313:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773245.985325:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce5880 x1862032663857024/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773245.985336:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985345:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3683 (0 0) 00000080:00200000:2.0:1775773245.985350:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773245.985379:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xe63:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773245.985386:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xe63:0x0], updating i_size 0 00000080:00200000:2.0:1775773245.985473:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xe63:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773245.985484:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xe63:0x0](ffff98296faec2c8) for lock 0x12ae90f9442e9e64 00000080:00010000:2.0:1775773245.985494:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xe63:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773245.985498:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9e64 from it ffffa414081d3df8 00010000:00010000:2.0:1775773245.985501:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773245.985509:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.985669:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773245.985678:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3248, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773245.985723:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773245.985748:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xe63:0x0]: losing layout lock 00010000:00010000:2.0:1775773245.986266:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773245.986274:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773245.986276:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6c00/0x12ae90f9442e9e64 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xe63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b5ff3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773245.986296:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773245.986304:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0e00 x1862032663857280/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773245.986326:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663857280:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773245.986456:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.291509:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.295902:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977aa9180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663865856:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.295928:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.295958:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.295972:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.295983:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.295993:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.296007:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.296015:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.296023:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.296034:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977aa9180 x1862032663865856/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.296045:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.296056:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5853 (0 0) 00000080:00200000:2.0:1775773246.296178:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.296224:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x16dd:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773246.296233:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x16dd:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.296329:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x16dd:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.296339:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x16dd:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9f05 00000080:00010000:2.0:1775773246.296349:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x16dd:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773246.296352:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9f05 from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.296356:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.296364:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.296392:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.296401:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7589, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.296445:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.296469:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x16dd:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.296521:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.296912:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.296916:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442e9f05 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x16dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b64fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.296944:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.296952:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0380 x1862032663866112/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.297416:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663866112:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.297515:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.302807:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663866112:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.302902:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x16dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.302912:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x16dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.302936:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x16dd:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773246.303015:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.303032:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6300, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.303037:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6300, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.303056:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6300,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.303137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9f0c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.303148:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9f0c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.303158:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442e9f0c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.303177:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663866240:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.303249:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773246.529537:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x10ff:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773246.529544:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x10ff:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.529632:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x10ff:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.529644:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x10ff:0x0](ffff98296faeba88) for lock 0x12ae90f9442e9f7c 00000080:00010000:2.0:1775773246.529655:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x10ff:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773246.529658:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9f7c from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.529662:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e9f7c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x10ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b68b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.529671:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e9f7c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b68b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.529696:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.529702:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4684, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.529736:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e9f7c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x10ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b68b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.530250:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x10ff:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.530289:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e9f7c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x10ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b68b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.530295:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.530296:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442e9f7c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x10ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b68b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.530313:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.530318:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3800 x1862032663872640/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.530338:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663872640:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.530452:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00010000:00010000:2.0:1775773246.755476:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755486:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755502:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755514:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755732:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755909:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.755915:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.755932:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9ce00 x1862032663878912/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.755948:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.756060:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6957 (0 0) 00000080:00200000:2.0:1775773246.756224:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.756265:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1b2d:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773246.756273:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1b2d:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.756366:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1b2d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.756376:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1b2d:0x0](ffff98296faee3c8) for lock 0x12ae90f9442e9ff3 00000080:00010000:2.0:1775773246.756387:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1b2d:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773246.756390:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9ff3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.756393:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.756403:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.756438:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.756446:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9796, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.756496:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.756520:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b2d:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.756718:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.756725:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.756727:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442e9ff3 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6c6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.756855:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.756865:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1f80 x1862032663879168/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.756891:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663879168:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.757335:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000080:00200000:2.0:1775773246.769353:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xe87:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773246.769361:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xe87:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.769448:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xe87:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.769459:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xe87:0x0](ffff98296faecb08) for lock 0x12ae90f9442e9ffa 00000080:00010000:2.0:1775773246.769469:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xe87:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773246.769472:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442e9ffa from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.769476:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e9ffa lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xe87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6ca3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.769485:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e9ffa lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6ca3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.770153:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.770164:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3421, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.770234:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e9ffa lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xe87:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6ca3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.770259:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xe87:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.770312:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e9ffa lrc: 1/0,0 mode: --/PR res: [0x240000403:0xe87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6ca3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.770319:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.770321:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442e9ffa lrc: 0/0,0 mode: --/PR res: [0x240000403:0xe87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6ca3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.770346:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.770356:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1500 x1862032663879552/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.770381:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663879552:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.770540:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773246.887318:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xe06:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.887328:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xe06:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.887354:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xe06:0x0](ffff982952597448) 00000080:00200000:2.0:1775773246.887435:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.887452:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9728, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.887457:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9728, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.887481:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9728,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.887661:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.887672:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.887682:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.887702:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663883904:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.887891:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.892153:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663883904:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.892179:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892207:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892216:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892231:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892241:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892254:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892263:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892272:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.892284:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f1c00 x1862032663883904/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.892297:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892308:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6873 (0 0) 00000080:00200000:2.0:1775773246.892314:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.892348:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1ad9:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.892354:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1ad9:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.892659:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1ad9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.892671:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1ad9:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea04e 00000080:00010000:2.0:1775773246.892682:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1ad9:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773246.892685:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea04e from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.892688:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.892697:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.892726:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.892734:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9728, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.892869:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.892893:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1ad9:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.892938:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.892945:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.892947:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea04e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1ad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6f43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.893201:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.893210:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1c00 x1862032663884160/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.893233:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884160:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.893362:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.897416:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884160:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.897509:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1ad9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.897519:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1ad9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.897558:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1ad9:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.897638:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.897656:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8908, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.897661:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8908, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.897681:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8908,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.897735:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.897757:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.897766:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.897785:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884288:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.897857:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.901594:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884288:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.901619:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901649:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901656:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901671:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901683:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901696:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901705:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901713:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.901722:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f0000 x1862032663884288/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.901734:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901756:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6513 (0 0) 00000080:00200000:2.0:1775773246.901763:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.901798:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1971:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773246.901806:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1971:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.901888:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1971:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.901897:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1971:0x0](ffff98296fae8908) for lock 0x12ae90f9442ea055 00000080:00010000:2.0:1775773246.901906:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1971:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773246.901910:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea055 from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.901913:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.901921:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.901947:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.901965:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8908, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.902010:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.902032:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1971:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.902110:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.902118:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.902121:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea055 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1971:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6f7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.902141:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.902148:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0000 x1862032663884544/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.902169:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884544:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.902243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.910575:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884544:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.910687:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1971:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.910696:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1971:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.910718:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1971:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773246.910813:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.910835:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.910840:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.910863:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8223,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.910919:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.910930:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.910940:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.910972:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884672:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.911053:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.914503:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884672:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.914541:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914572:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: --/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914580:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: --/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914595:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914606:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914619:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914627:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914634:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.914645:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f2d80 x1862032663884672/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.914657:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914667:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6170 (0 0) 00000080:00200000:2.0:1775773246.914674:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.914705:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x181a:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.914712:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x181a:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.914830:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x181a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.914842:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x181a:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea05c 00000080:00010000:2.0:1775773246.914854:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x181a:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773246.914858:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea05c from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.914861:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.914871:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.914907:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.914915:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8223, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.914979:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.915003:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x181a:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.915503:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 1/0,0 mode: --/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.915512:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.915515:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea05c lrc: 0/0,0 mode: --/PR res: [0x200000403:0x181a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6fb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.915539:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.915546:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2d80 x1862032663884928/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.915569:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.915642:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.920191:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663884928:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.920283:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x181a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.920294:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x181a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.920317:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x181a:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.920400:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.920419:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.920425:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.920456:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6041,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.920524:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.920548:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.920561:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.920584:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885056:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.920659:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.924412:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885056:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.924448:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924477:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924485:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924500:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924511:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924523:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924542:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924549:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.924559:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f3b80 x1862032663885056/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.924571:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.924581:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5079 (0 0) 00000080:00200000:2.0:1775773246.924588:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.924635:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x13d7:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773246.924643:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x13d7:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.924735:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x13d7:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.925058:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x13d7:0x0](ffff98296fae8908) for lock 0x12ae90f9442ea063 00000080:00010000:2.0:1775773246.925101:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x13d7:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773246.925105:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea063 from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.925109:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.925119:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.925154:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.925162:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6041, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.925210:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.925234:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x13d7:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.925281:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.925289:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.925291:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea063 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x13d7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b6feb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.925313:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.925321:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3b80 x1862032663885312/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.925351:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885312:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.925431:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.929195:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885312:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.929278:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x13d7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.929287:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x13d7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.929308:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x13d7:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773246.929386:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.929404:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8696, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.929409:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8696, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.929431:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8696,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.929491:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.929502:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.929512:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.929532:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885440:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.929617:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.934509:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885440:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.934551:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934581:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934589:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934605:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934617:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934630:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934638:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934645:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.934657:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f0a80 x1862032663885440/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.934668:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934677:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6407 (0 0) 00000080:00200000:2.0:1775773246.934684:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.934715:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1907:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.934722:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1907:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.934823:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1907:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.934835:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1907:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea06a 00000080:00010000:2.0:1775773246.934846:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1907:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773246.934850:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea06a from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.934853:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.934862:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.934894:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.934902:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8696, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.934949:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.934973:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1907:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.935017:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.935024:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.935027:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ea06a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1907:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b7023 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.935048:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.935055:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0a80 x1862032663885696/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.935113:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885696:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.935199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.939771:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885696:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773246.939864:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1907:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773246.939875:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1907:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773246.939902:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1907:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773246.940007:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773246.940028:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9817, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773246.940033:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9817, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773246.940057:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9817,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773246.940185:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.940196:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.940206:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773246.940234:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885824:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773246.940318:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773246.943691:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663885824:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773246.943719:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943867:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943876:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943893:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943905:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943919:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943927:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.943934:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773246.943946:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f3100 x1862032663885824/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773287 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773246.944041:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.944055:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6967 (0 0) 00000080:00200000:2.0:1775773246.944157:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773246.944198:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1b37:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773246.944206:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1b37:0x0], updating i_size 0 00000080:00200000:2.0:1775773246.944297:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1b37:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773246.944308:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1b37:0x0](ffff98296fae8908) for lock 0x12ae90f9442ea071 00000080:00010000:2.0:1775773246.944318:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1b37:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773246.944321:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea071 from it ffffa414081d3df8 00010000:00010000:2.0:1775773246.944325:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773246.944333:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.944366:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773246.944374:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9817, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773246.944427:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773246.944451:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b37:0x0]: losing layout lock 00010000:00010000:2.0:1775773246.944503:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773246.944509:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773246.944512:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ea071 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b705b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773246.944531:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773246.944537:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3100 x1862032663886080/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773246.944558:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663886080:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773246.944692:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773247.040838:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xe3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.040854:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xe3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.040883:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xe3d:0x0](ffff9829525900c8) 00000080:00200000:2.0:1775773247.041283:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.041305:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2567, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.041311:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2567, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.041334:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2567,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.041404:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea0b7 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.041416:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea0b7 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.041427:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea0b7 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.041450:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663889664:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.041707:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773247.106408:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.106430:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9637, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.106437:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9637, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.106464:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9637,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.106522:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea0e8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.106589:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea0e8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.106600:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea0e8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.106623:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663892352:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.106757:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773247.199924:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x158b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773247.199940:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x158b:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.200319:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x158b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.200332:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x158b:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea119 00000080:00010000:2.0:1775773247.200344:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x158b:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773247.200349:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea119 from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.200353:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea119 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x158b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b759b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.200364:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea119 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x158b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b759b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.200402:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.200410:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7012, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.200460:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea119 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x158b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b759b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.200487:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x158b:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.200536:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea119 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x158b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b759b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.200544:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.200546:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea119 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x158b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b759b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.200575:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.200587:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2300 x1862032663895296/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.200618:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663895296:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.200962:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.205979:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663895296:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773247.206621:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x158b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.206632:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x158b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.206655:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x158b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773247.206918:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.206937:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1441, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.206942:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1441, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.207235:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1441,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.207298:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442ea120 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.207308:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442ea120 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.207318:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442ea120 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.207341:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663895424:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.207485:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00010000:00010000:2.0:1775773247.280386:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280393:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280405:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280414:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280424:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280430:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280434:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773247.280442:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ec0700 x1862032663897344/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773247.280451:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280458:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6731 (0 0) 00000080:00200000:2.0:1775773247.280463:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773247.280495:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1a4b:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773247.280501:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1a4b:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.280630:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1a4b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.280638:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1a4b:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea143 00000080:00010000:2.0:1775773247.280647:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1a4b:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773247.280649:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea143 from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.280652:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.280658:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.280684:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.280692:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9345, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.280734:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.280757:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1a4b:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.280801:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.280808:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.280811:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ea143 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1a4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b76eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.280833:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.280840:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3100 x1862032663897600/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.280862:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663897600:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.281001:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.349267:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900032:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773247.349293:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349317:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349323:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349332:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349339:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349347:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349351:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349356:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773247.349363:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9aa00 x1862032663900032/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773247.349372:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349379:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6989 (0 0) 00000080:00200000:2.0:1775773247.349384:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773247.349411:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1b4d:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773247.349417:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1b4d:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.349488:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1b4d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.349494:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1b4d:0x0](ffff98296faedb88) for lock 0x12ae90f9442ea174 00000080:00010000:2.0:1775773247.349501:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1b4d:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773247.349503:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea174 from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.349505:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.349512:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.349539:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.349547:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9860, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.349591:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.349616:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1b4d:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.349655:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.349661:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.349663:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea174 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b7873 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.349687:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.349694:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0a80 x1862032663900288/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.349717:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900288:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.350233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.353407:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900288:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773247.353483:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1b4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.353491:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1b4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.353511:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1b4d:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773247.353647:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.353679:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8559, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.353682:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8559, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.353701:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8559,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.353752:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.353762:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.353772:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.353792:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900416:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.353854:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.358170:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900416:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773247.358203:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358244:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358255:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358266:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358278:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358290:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358298:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358306:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773247.358318:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f3b80 x1862032663900416/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773247.358329:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358340:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6338 (0 0) 00000080:00200000:2.0:1775773247.358348:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773247.358384:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x18c2:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773247.358422:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x18c2:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.358543:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x18c2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.358555:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x18c2:0x0](ffff98296faea1c8) for lock 0x12ae90f9442ea17b 00000080:00010000:2.0:1775773247.358567:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x18c2:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773247.358570:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea17b from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.358574:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.358583:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.358615:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.358625:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8559, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.358683:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.358700:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x18c2:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.358735:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 1/0,0 mode: --/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.358741:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.358744:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea17b lrc: 0/0,0 mode: --/PR res: [0x200000403:0x18c2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b78ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.358771:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.358778:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f3b80 x1862032663900672/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.358802:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900672:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.358875:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.362263:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900672:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773247.362352:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x18c2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.362363:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x18c2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.362388:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x18c2:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773247.362462:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.362480:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1847, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.362484:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1847, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.362502:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1847,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.362555:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.362566:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.362576:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.362597:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900800:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.363561:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.365882:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663900800:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773247.365916:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.365963:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.365972:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.365984:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.365997:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.366011:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.366020:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.366029:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773247.366041:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f2d80 x1862032663900800/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773247.366104:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.366119:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2932 (0 0) 00000080:00200000:2.0:1775773247.366125:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773247.366170:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xb74:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773247.366178:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xb74:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.366264:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xb74:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.366275:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xb74:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea182 00000080:00010000:2.0:1775773247.366287:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xb74:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773247.366291:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea182 from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.366294:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.366303:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.366336:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.366345:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1847, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.366390:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.366412:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xb74:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.366453:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.366460:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.366463:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea182 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b78e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.366488:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.366495:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2d80 x1862032663901056/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.366516:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663901056:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.366896:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.371248:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663901056:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773247.371327:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xb74:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.371335:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xb74:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.371352:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xb74:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773247.371409:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.371420:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2495, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.371424:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2495, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.371440:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2495,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.371482:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.371491:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.371499:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.371512:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663901184:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.371582:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000100:00100000:2.0:1775773247.375237:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663901184:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773247.375262:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375285:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375291:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375300:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375307:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375316:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375321:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375326:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773247.375333:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f0000 x1862032663901184/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773247.375341:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375349:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3256 (0 0) 00000080:00200000:2.0:1775773247.375354:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773247.375377:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xcb8:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773247.375382:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xcb8:0x0], updating i_size 0 00000080:00200000:2.0:1775773247.375450:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xcb8:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773247.375456:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xcb8:0x0](ffff98296faea1c8) for lock 0x12ae90f9442ea189 00000080:00010000:2.0:1775773247.375463:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xcb8:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773247.375466:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea189 from it ffffa414081d3df8 00010000:00010000:2.0:1775773247.375468:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.375474:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.375496:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773247.375502:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2495, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773247.375546:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773247.375569:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xcb8:0x0]: losing layout lock 00010000:00010000:2.0:1775773247.375611:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773247.375617:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773247.375619:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea189 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xcb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b791b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773247.375638:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773247.375643:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0000 x1862032663901440/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773247.375660:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663901440:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773247.375727:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773247.793207:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1b25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773247.793219:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1b25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773247.793245:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1b25:0x0](ffff98297d9a2a08) 00000080:00200000:2.0:1775773247.793334:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773247.793367:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9776, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773247.793372:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9776, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773247.793395:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9776,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773247.793457:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea270 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.793468:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea270 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773247.793479:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442ea270 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773247.793501:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663913856:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773247.793902:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773248.030303:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 4/1,0 mode: --/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030319:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 4/1,0 mode: --/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030332:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030344:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030360:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030368:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030375:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.030387:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965188e00 x1862032663920768/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.030400:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.030412:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2577 (0 0) 00000080:00200000:2.0:1775773248.030418:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.030446:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xa11:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773248.030454:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xa11:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.031233:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xa11:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.031252:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xa11:0x0](ffff98296faeaa08) for lock 0x12ae90f9442ea2ee 00000080:00010000:2.0:1775773248.031264:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xa11:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773248.031267:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea2ee from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.031270:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.031281:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.031314:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.031321:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1137, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.031369:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.031390:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xa11:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.031439:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 1/0,0 mode: --/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.031445:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.031447:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ea2ee lrc: 0/0,0 mode: --/PR res: [0x240000403:0xa11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8443 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.031470:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.031477:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0380 x1862032663921024/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.031502:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663921024:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.031604:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.101056:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1537, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.101222:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442ea311 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xad9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b855b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.101250:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xad9:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.101307:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442ea311 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b855b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.101314:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.101316:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442ea311 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xad9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b855b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.101346:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.101357:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0e00 x1862032663922944/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.101385:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663922944:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.101512:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.118384:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923328:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773248.118472:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1a61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.118483:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1a61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.118507:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1a61:0x0](ffff98297d9a3a88) 00000080:00200000:2.0:1775773248.119240:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.119263:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f873, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.119269:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f873, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.119295:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f873,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.119378:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.119389:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.119402:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.119430:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923456:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.121515:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.124268:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923456:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.124297:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124331:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124341:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124353:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124365:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124380:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124395:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124402:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.124411:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829737f2680 x1862032663923456/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.124423:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124433:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2445 (0 0) 00000080:00200000:2.0:1775773248.124440:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.124478:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x98d:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773248.124486:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x98d:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.124590:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x98d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.124602:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x98d:0x0](ffff98296faedb88) for lock 0x12ae90f9442ea31f 00000080:00010000:2.0:1775773248.124612:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x98d:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773248.124616:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea31f from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.124619:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.124627:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.124670:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.124680:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f873, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.124739:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.124763:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x98d:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.124812:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.124818:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.124821:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea31f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x98d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b85cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.124844:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.124850:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2680 x1862032663923712/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.124874:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.124939:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.128810:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829737f2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923712:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773248.128909:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x98d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.128920:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x98d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.128961:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x98d:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773248.129106:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.129129:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4393, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.129136:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4393, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.129166:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4393,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.129233:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ea326 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.129248:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ea326 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.129258:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ea326 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.129279:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663923840:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.129342:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.220414:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1e40 going to sleep for 36 seconds 00010000:00010000:2.0:1775773248.244403:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244414:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244428:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244441:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244461:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244470:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244478:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.244492:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce6680 x1862032663927296/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.244505:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244516:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5181 (0 0) 00000080:00200000:2.0:1775773248.244522:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.244795:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x143d:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773248.244805:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x143d:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.244890:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x143d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.244902:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x143d:0x0](ffff98296faedb88) for lock 0x12ae90f9442ea365 00000080:00010000:2.0:1775773248.244912:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x143d:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773248.244915:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea365 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.244919:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.244928:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.245062:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.245164:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6345, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.245211:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.245232:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x143d:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.245279:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.245286:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.245288:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea365 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x143d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b87fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.245309:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.245317:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1180 x1862032663927552/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.245340:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663927552:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.245475:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.328301:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x9be:0x0](ffff982952590908) 00000080:00200000:2.0:1775773248.328310:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x9be:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.328390:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x9be:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.328402:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x9be:0x0](ffff982952590908) for lock 0x12ae90f9442ea39d 00000080:00010000:2.0:1775773248.328412:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x9be:0x0](ffff982952590908) 00000080:00010000:2.0:1775773248.328416:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea39d from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.328419:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea39d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x9be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b89bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.328428:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea39d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b89bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.328457:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.328465:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f970, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.328512:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea39d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x9be:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b89bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.328552:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x9be:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.328595:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea39d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x9be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b89bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.328602:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.328604:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea39d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x9be:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b89bb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.328626:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.328634:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1880 x1862032663930624/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.328656:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663930624:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.328781:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773248.386367:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386377:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386389:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386401:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386413:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386421:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386428:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.386439:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91f80 x1862032663932288/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.386451:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386460:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3165 (0 0) 00000080:00200000:2.0:1775773248.386466:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.386501:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xc5d:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773248.386508:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xc5d:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.386677:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xc5d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.386690:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xc5d:0x0](ffff98296fae8908) for lock 0x12ae90f9442ea3c0 00000080:00010000:2.0:1775773248.386701:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xc5d:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773248.386705:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea3c0 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.386708:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.386807:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.386844:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.386854:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2312, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.386907:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.387044:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc5d:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.387231:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.387240:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.387243:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ea3c0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8ad3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.387265:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.387274:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f0700 x1862032663932544/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.387298:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663932544:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.387427:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.436993:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1149:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.437004:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1149:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.437026:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1149:0x0](ffff982952590908) 00000080:00200000:2.0:1775773248.437160:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.437179:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5739, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.437184:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5739, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.437205:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5739,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.437263:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442ea3dc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.437274:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442ea3dc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.437285:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442ea3dc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.437308:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663933824:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.437389:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.512629:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1251:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.512655:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1251:0x0](ffff98296fae63c8) for lock 0x12ae90f9442ea40d 00000080:00010000:2.0:1775773248.512667:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1251:0x0](ffff98296fae63c8) 00000080:00010000:2.0:1775773248.512670:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea40d from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.512674:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4000/0x12ae90f9442ea40d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1251:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b8d3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.512686:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4000/0x12ae90f9442ea40d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1251:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b8d3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.512884:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.512894:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5360, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.513029:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4000/0x12ae90f9442ea40d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1251:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b8d3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.513095:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1251:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.513145:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4000/0x12ae90f9442ea40d lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1251:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8d3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.513153:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.513155:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4000/0x12ae90f9442ea40d lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1251:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b8d3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.513188:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.513198:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f1f80 x1862032663936768/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.513225:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663936768:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.513314:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.733454:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1988:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773248.733462:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1988:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.733656:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1988:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.733670:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1988:0x0](ffff98296faecb08) for lock 0x12ae90f9442ea492 00000080:00010000:2.0:1775773248.733681:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1988:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773248.733685:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea492 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.733688:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ea492 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1988:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.733698:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ea492 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1988:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.733743:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.733751:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8955, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.733795:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ea492 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1988:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.733817:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1988:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.733859:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ea492 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1988:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.733871:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.733873:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ea492 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1988:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.733895:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.733903:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829737f2a00 x1862032663944064/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.734358:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829737f2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663944064:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.734511:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773248.802311:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f754, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.802391:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442ea4b5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x952:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b927b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.802417:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x952:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.802475:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442ea4b5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x952:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b927b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.802484:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.802486:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442ea4b5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x952:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b927b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.802512:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.802521:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4000 x1862032663945984/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.802562:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663945984:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.802663:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.945485:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.945830:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663949952:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.945855:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.945888:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.945897:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.945907:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946205:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946220:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946227:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946233:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.946243:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94380 x1862032663949952/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.946259:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946269:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3536 (0 0) 00000080:00200000:2.0:1775773248.946275:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.946311:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xdd0:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773248.946318:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xdd0:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.946400:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xdd0:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.946412:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xdd0:0x0](ffff98296faee3c8) for lock 0x12ae90f9442ea502 00000080:00010000:2.0:1775773248.946421:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xdd0:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773248.946425:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea502 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.946428:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.946437:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.946466:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.946474:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3054, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.946518:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.946787:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xdd0:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.946831:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.946838:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.946842:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ea502 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b94e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.946863:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.946870:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6300 x1862032663950208/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.946891:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950208:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.947204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.957398:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950336:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.957425:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957456:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957464:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957474:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957484:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957497:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957508:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957515:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.957525:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982974b2fb80 x1862032663950336/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.957552:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.957562:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3430 (0 0) 00000080:00200000:2.0:1775773248.957568:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.957596:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xd66:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773248.957602:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xd66:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.957691:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xd66:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.959038:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xd66:0x0](ffff98296faeba88) for lock 0x12ae90f9442ea509 00000080:00010000:2.0:1775773248.959057:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xd66:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773248.959061:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea509 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.959078:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.959093:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.959133:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.959143:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2742, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.959216:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.959240:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xd66:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.959283:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.959290:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.959293:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442ea509 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xd66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b951b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.959315:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.959324:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6a00 x1862032663950592/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.959346:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950592:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.959594:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.965490:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950592:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773248.965591:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xd66:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.965601:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xd66:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.965625:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xd66:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773248.965906:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.965931:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3705, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.965936:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3705, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.965959:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3705,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.966020:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.966046:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.966056:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.966097:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950720:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.966218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.970428:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950720:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.970455:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970484:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970492:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970503:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970514:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970592:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970607:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970614:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.970625:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4e00 x1862032663950720/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.970786:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.970796:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3911 (0 0) 00000080:00200000:2.0:1775773248.970803:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.970834:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xf47:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773248.970841:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xf47:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.971018:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xf47:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.971041:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xf47:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea510 00000080:00010000:2.0:1775773248.971052:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xf47:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773248.971056:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea510 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.971059:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.971289:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.971323:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.971334:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3705, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.971386:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.971409:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xf47:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.971450:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.971456:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.971459:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea510 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xf47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.971479:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.971486:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4e00 x1862032663950976/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.971511:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950976:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.971900:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.975892:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663950976:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773248.975984:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xf47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.975994:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xf47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.976015:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xf47:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773248.976112:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.976127:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8803, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.976131:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8803, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.976151:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8803,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.976202:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.976215:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.976224:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.976241:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951104:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.976308:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.980894:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951104:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.980932:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.980964:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.980973:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.980986:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.980998:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.981011:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.981034:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.981043:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.981055:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032663951104/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.981114:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.981125:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6460 (0 0) 00000080:00200000:2.0:1775773248.981131:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.981166:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x193c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773248.981173:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x193c:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.981251:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x193c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.981260:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x193c:0x0](ffff98296faeba88) for lock 0x12ae90f9442ea517 00000080:00010000:2.0:1775773248.981270:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x193c:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773248.981273:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea517 from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.981276:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.981286:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.981314:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.981323:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8803, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.981363:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.981382:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x193c:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.981427:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.981434:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.981437:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ea517 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x193c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b958b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.981457:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.981463:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5180 x1862032663951360/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.981484:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951360:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.981560:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.986121:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951360:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773248.986207:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x193c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773248.986217:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x193c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773248.986238:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x193c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773248.986314:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773248.986329:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7057, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773248.986333:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7057, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773248.986354:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7057,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773248.986408:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.986421:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.986431:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773248.986452:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951488:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773248.986524:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773248.992857:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951488:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773248.992882:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.992915:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993033:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993045:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993056:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993154:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993169:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993177:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773248.993187:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5f80 x1862032663951488/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773289 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773248.993200:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993210:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5587 (0 0) 00000080:00200000:2.0:1775773248.993216:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773248.993246:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x15d3:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773248.993253:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x15d3:0x0], updating i_size 0 00000080:00200000:2.0:1775773248.993331:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x15d3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773248.993340:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x15d3:0x0](ffff98296faeba88) for lock 0x12ae90f9442ea51e 00000080:00010000:2.0:1775773248.993349:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x15d3:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773248.993352:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea51e from it ffffa414081d3df8 00010000:00010000:2.0:1775773248.993355:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773248.993364:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.993393:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773248.993402:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7057, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773248.993446:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773248.993469:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x15d3:0x0]: losing layout lock 00010000:00010000:2.0:1775773248.993516:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773248.993522:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773248.993525:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442ea51e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x15d3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b95c3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773248.993912:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773248.994014:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5f80 x1862032663951744/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773248.994049:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663951744:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773248.994262:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773249.111479:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442ea556 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12e0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9783 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.111495:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.111499:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442ea556 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12e0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9783 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.111527:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.111538:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5c00 x1862032663954816/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.111562:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663954816:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.111820:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000080:00200000:2.0:1775773249.290219:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1639:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.290265:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1639:0x0](ffff98296fae6c08) for lock 0x12ae90f9442ea5b8 00000080:00010000:2.0:1775773249.290280:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1639:0x0](ffff98296fae6c08) 00000080:00010000:2.0:1775773249.290284:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea5b8 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.290289:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4a00/0x12ae90f9442ea5b8 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1639:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9a93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.290302:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4a00/0x12ae90f9442ea5b8 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1639:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9a93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.290337:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.290344:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7260, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.290398:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4a00/0x12ae90f9442ea5b8 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1639:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9a93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.290422:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1639:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.290464:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4a00/0x12ae90f9442ea5b8 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1639:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9a93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.290471:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.290473:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4a00/0x12ae90f9442ea5b8 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1639:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9a93 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.290501:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.290514:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6d80 x1862032663960192/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.290551:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663960192:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.290649:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.297566:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa66c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.406546:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db19c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773249.418946:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1820:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.418959:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1820:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.418988:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1820:0x0](ffff982952593248) 00000080:00200000:2.0:1775773249.419123:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.419143:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7946, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.419148:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7946, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.419169:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7946,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.419223:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.419233:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.419242:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.419264:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663963776:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.419345:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.423600:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663963776:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.423649:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423684:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: --/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423706:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: --/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423717:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423730:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423742:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423750:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423757:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.423768:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4700 x1862032663963776/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.423780:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423790:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5982 (0 0) 00000080:00200000:2.0:1775773249.423796:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.423828:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x175e:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.423835:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x175e:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.423909:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x175e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.423930:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x175e:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea5fe 00000080:00010000:2.0:1775773249.423939:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x175e:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773249.423942:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea5fe from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.423945:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.423953:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.423980:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.423988:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7946, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.424043:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.424354:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x175e:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.424406:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 1/0,0 mode: --/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.424413:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.424415:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea5fe lrc: 0/0,0 mode: --/PR res: [0x240000403:0x175e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9cc3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.424439:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.424446:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4700 x1862032663964032/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.424467:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964032:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.424549:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.429351:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964032:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.429440:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x175e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.429449:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x175e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.429469:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x175e:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.429830:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.429850:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6577, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.429855:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6577, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.429879:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6577,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.430045:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.430055:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.430308:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.430332:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964160:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.430405:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.435010:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964160:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.435052:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435112:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435125:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435136:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435147:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435160:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435168:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435175:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.435186:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5500 x1862032663964160/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.435198:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435207:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5347 (0 0) 00000080:00200000:2.0:1775773249.435213:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.435244:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x14e3:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.435251:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x14e3:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.435334:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x14e3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.435346:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x14e3:0x0](ffff98296fae9148) for lock 0x12ae90f9442ea605 00000080:00010000:2.0:1775773249.435357:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x14e3:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773249.435361:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea605 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.435365:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.435375:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.435411:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.435420:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6577, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.435467:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.435491:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x14e3:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.435538:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.435545:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.435547:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea605 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x14e3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9cfb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.435570:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.435578:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5500 x1862032663964416/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.435607:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964416:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.435699:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.441376:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964416:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.441460:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x14e3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.441469:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x14e3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.441493:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x14e3:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.441576:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.441591:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1831, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.441596:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1831, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.441627:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1831,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.441703:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.441715:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.441725:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.441748:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964544:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.441825:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.448379:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964544:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.448407:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448440:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448453:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448463:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448475:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448489:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448497:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448503:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.448513:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7100 x1862032663964544/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.448526:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448537:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2974 (0 0) 00000080:00200000:2.0:1775773249.448544:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.448585:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xb9e:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.448594:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xb9e:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.448718:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xb9e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.448729:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xb9e:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea60c 00000080:00010000:2.0:1775773249.448739:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xb9e:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773249.448743:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea60c from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.448746:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.448756:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.448787:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.448797:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1831, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.448844:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.448865:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb9e:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.448909:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.448916:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.448919:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea60c lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9d33 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.448941:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.448948:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7100 x1862032663964800/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.448972:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964800:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.449095:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.456058:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964800:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.456180:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xb9e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.456189:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xb9e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.456212:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xb9e:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.456289:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.456307:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.456311:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.456331:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9658,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.456388:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.456397:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.456407:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.456429:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964928:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.456505:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.462130:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663964928:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.462156:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462187:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462200:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462211:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462222:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462235:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462242:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462250:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.462261:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7800 x1862032663964928/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.462274:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462283:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6838 (0 0) 00000080:00200000:2.0:1775773249.462290:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.462317:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1ab6:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.462325:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1ab6:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.462413:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1ab6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.462424:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1ab6:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea613 00000080:00010000:2.0:1775773249.462432:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1ab6:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773249.462435:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea613 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.462439:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.462449:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.462480:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.462488:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9658, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.462531:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.462554:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1ab6:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.462599:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.462605:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.462608:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea613 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1ab6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9d6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.462642:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.462649:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7800 x1862032663965184/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.462673:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965184:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.462747:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.468392:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965184:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.468484:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1ab6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.468496:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1ab6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.468538:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1ab6:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.468647:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.468666:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7438, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.468671:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7438, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.468692:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7438,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.468760:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.468770:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.468781:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.468800:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965312:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.468871:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.472876:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965312:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.472903:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.472938:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.472950:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.472961:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.472971:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.472995:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.473003:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.473010:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.473029:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6680 x1862032663965312/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.473043:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.473054:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5728 (0 0) 00000080:00200000:2.0:1775773249.473061:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.473131:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1660:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.473139:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1660:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.473228:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1660:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.473240:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1660:0x0](ffff98296fae9148) for lock 0x12ae90f9442ea61a 00000080:00010000:2.0:1775773249.473250:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1660:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773249.473254:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea61a from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.473257:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.473267:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.473298:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.473307:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7438, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.473349:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.473369:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1660:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.473412:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.473418:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.473421:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea61a lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1660:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9da3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.473442:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.473449:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6680 x1862032663965568/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.473470:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965568:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.473545:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.478412:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965568:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.478493:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1660:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.478502:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1660:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.478523:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1660:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.478590:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.478605:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4222, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.478610:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4222, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.478644:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4222,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.478699:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.478711:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.478722:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.478740:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965696:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.478801:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.483687:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965696:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.483713:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483745:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483759:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483771:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483782:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483794:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483802:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483809:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.483821:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7b80 x1862032663965696/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.483834:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.483844:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4120 (0 0) 00000080:00200000:2.0:1775773249.483851:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.483888:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1018:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.483894:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1018:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.483996:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1018:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.484010:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1018:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea621 00000080:00010000:2.0:1775773249.484030:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1018:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773249.484034:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea621 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.484038:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.484047:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.484130:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.484138:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4222, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.484185:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.484208:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1018:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.484249:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.484256:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.484258:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea621 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1018:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9ddb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.484277:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.484284:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7b80 x1862032663965952/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.484306:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965952:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.484496:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.487688:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663965952:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.487928:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1018:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.487939:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1018:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.487963:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1018:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.488961:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.489205:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2129, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.489212:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2129, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.489236:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2129,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.489297:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.489309:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.489322:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.489351:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.489507:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.492548:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966080:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.492572:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492601:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492615:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492626:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492635:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492646:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492654:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492661:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.492671:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4a80 x1862032663966080/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.492684:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.492870:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3073 (0 0) 00000080:00200000:2.0:1775773249.492879:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.492917:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xc01:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.492924:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xc01:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.493012:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xc01:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.493031:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xc01:0x0](ffff98296fae9148) for lock 0x12ae90f9442ea628 00000080:00010000:2.0:1775773249.493041:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xc01:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773249.493045:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea628 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.493048:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.493057:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.493114:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.493122:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2129, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.493169:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.493189:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc01:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.493228:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.493235:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.493237:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea628 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc01:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e13 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.493256:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.493262:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4a80 x1862032663966336/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.493283:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966336:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.493355:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.497428:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966336:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.497517:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xc01:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.497528:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xc01:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.497554:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xc01:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.497632:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.497661:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8148, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.497667:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8148, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.497702:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8148,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.497760:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.497770:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.497780:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.497799:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966464:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.497868:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.501608:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966464:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.501654:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501681:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501705:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501717:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501728:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501740:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501749:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501756:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.501766:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7480 x1862032663966464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.501778:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.501788:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6133 (0 0) 00000080:00200000:2.0:1775773249.501794:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.501828:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x17f5:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.501835:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x17f5:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.502305:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x17f5:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.502318:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x17f5:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea62f 00000080:00010000:2.0:1775773249.502329:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x17f5:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773249.502332:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea62f from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.502335:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.502344:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.502376:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.502385:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8148, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.502433:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.502455:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x17f5:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.502503:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 1/0,0 mode: --/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.502510:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.502513:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ea62f lrc: 0/0,0 mode: --/PR res: [0x200000403:0x17f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e4b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.502538:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.502545:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7480 x1862032663966720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.502568:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966720:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.502845:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.511340:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966720:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773249.511438:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x17f5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773249.511450:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x17f5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773249.511472:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x17f5:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773249.511547:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773249.511565:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9920, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773249.511569:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9920, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773249.511591:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9920,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773249.511659:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.511670:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.511680:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773249.511711:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773249.511783:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773249.517655:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663966848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773249.517681:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517721:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517735:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517745:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517755:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517766:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517774:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517780:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773249.517789:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4380 x1862032663966848/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773290 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773249.517800:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517809:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6969 (0 0) 00000080:00200000:2.0:1775773249.517815:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773249.517844:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1b39:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773249.517851:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1b39:0x0], updating i_size 0 00000080:00200000:2.0:1775773249.517949:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1b39:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773249.517960:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1b39:0x0](ffff98296fae9148) for lock 0x12ae90f9442ea636 00000080:00010000:2.0:1775773249.517970:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1b39:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773249.517973:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea636 from it ffffa414081d3df8 00010000:00010000:2.0:1775773249.517976:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773249.517984:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.518010:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773249.518030:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9920, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773249.518114:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773249.518138:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1b39:0x0]: losing layout lock 00010000:00010000:2.0:1775773249.518186:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773249.518193:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773249.518196:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ea636 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1b39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2b9e83 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773249.518218:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773249.518225:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4380 x1862032663967104/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773249.518247:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663967104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773249.518319:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773250.041246:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3058, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773250.041252:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3058, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773250.041274:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3058,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773250.041342:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ea740 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.041355:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ea740 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.041366:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ea740 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773250.041391:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663982080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773250.041491:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000080:00200000:2.0:1775773250.130265:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1276:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773250.130273:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1276:0x0], updating i_size 0 00000080:00200000:2.0:1775773250.130353:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1276:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773250.130364:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1276:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ea771 00000080:00010000:2.0:1775773250.130375:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1276:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773250.130378:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea771 from it ffffa414081d3df8 00010000:00010000:2.0:1775773250.130382:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ea771 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1276:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ba85b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.130391:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ea771 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1276:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ba85b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.130423:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773250.130431:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5334, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773250.130475:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ea771 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1276:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ba85b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.130500:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1276:0x0]: losing layout lock 00010000:00010000:2.0:1775773250.130888:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ea771 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1276:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ba85b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773250.130896:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773250.130990:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ea771 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1276:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ba85b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773250.131029:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773250.131039:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6bb80 x1862032663985024/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773250.131157:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663985024:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773250.131306:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000080:00200000:2.0:1775773250.164047:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xb1c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773250.164176:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xb1c:0x0](ffff982952595348) for lock 0x12ae90f9442ea786 00000080:00010000:2.0:1775773250.164189:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xb1c:0x0](ffff982952595348) 00000080:00010000:2.0:1775773250.164192:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea786 from it ffffa414081d3df8 00010000:00010000:2.0:1775773250.164196:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ea786 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2ba903 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.164208:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ea786 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2ba903 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.164242:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773250.164250:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1670, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773250.164298:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ea786 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb1c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2ba903 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.164324:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xb1c:0x0]: losing layout lock 00010000:00010000:2.0:1775773250.164370:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ea786 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ba903 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773250.164377:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773250.164380:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ea786 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2ba903 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773250.164407:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773250.164420:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b480 x1862032663986176/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773250.164445:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663986176:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773250.164745:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773250.315523:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442ea7da lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1141:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2baba3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773250.315629:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773250.315632:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7c00/0x12ae90f9442ea7da lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1141:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2baba3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773250.315661:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773250.315671:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff98296f042680 x1862032663990784/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773250.315860:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98296f042680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032663990784:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773250.316032:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773250.753263:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773250.753284:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7592, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773250.753289:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7592, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773250.753317:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7592,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773250.753394:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea8c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.753405:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea8c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.753415:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ea8c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773250.753436:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664003584:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773250.753510:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773250.815821:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54aa80 going to sleep for 36 seconds 00010000:00010000:2.0:1775773250.857628:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857648:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857660:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857763:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857783:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857793:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857800:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773250.857816:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ce2a00 x1862032664005888/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773250.857829:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.857840:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3771 (0 0) 00000080:00200000:2.0:1775773250.857847:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773250.857878:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xebb:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773250.857984:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xebb:0x0], updating i_size 0 00000080:00200000:2.0:1775773250.858227:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xebb:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773250.858240:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xebb:0x0](ffff98296fae80c8) for lock 0x12ae90f9442ea8f2 00000080:00010000:2.0:1775773250.858250:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xebb:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773250.858254:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea8f2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773250.858257:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.858265:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.858297:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773250.858306:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3524, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773250.858349:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.858371:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xebb:0x0]: losing layout lock 00010000:00010000:2.0:1775773250.858433:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773250.858440:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773250.858443:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ea8f2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xebb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bb463 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773250.858465:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773250.858473:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6ad80 x1862032664006144/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773250.858498:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664006144:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773250.858601:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000080:00200000:2.0:1775773250.908770:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1112:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773250.908791:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1112:0x0](ffff98296fae6c08) for lock 0x12ae90f9442ea90e 00000080:00010000:2.0:1775773250.908802:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1112:0x0](ffff98296fae6c08) 00000080:00010000:2.0:1775773250.908806:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea90e from it ffffa414081d3df8 00010000:00010000:2.0:1775773250.908810:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4c00/0x12ae90f9442ea90e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bb543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773250.908818:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4c00/0x12ae90f9442ea90e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bb543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.908849:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773250.908858:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4723, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773250.908912:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4c00/0x12ae90f9442ea90e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1112:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bb543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773250.908934:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1112:0x0]: losing layout lock 00010000:00010000:2.0:1775773250.908976:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4c00/0x12ae90f9442ea90e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bb543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773250.908984:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773250.908986:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4c00/0x12ae90f9442ea90e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1112:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bb543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773250.909020:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773250.909027:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68700 x1862032664007680/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773250.909050:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664007680:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773250.909151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000080:00200000:2.0:1775773251.184292:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xa5a:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773251.184299:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xa5a:0x0], updating i_size 0 00000080:00200000:2.0:1775773251.184378:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xa5a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773251.184388:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xa5a:0x0](ffff98296faeec08) for lock 0x12ae90f9442ea9d2 00000080:00010000:2.0:1775773251.184399:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xa5a:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773251.184402:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ea9d2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773251.184406:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea9d2 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bbb63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.184415:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea9d2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bbb63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.184452:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773251.184459:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1183, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.184498:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea9d2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa5a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bbb63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.184521:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa5a:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.184782:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea9d2 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bbb63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.184789:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.184792:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ea9d2 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bbb63 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.184816:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.184823:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6a680 x1862032664018432/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.184844:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664018432:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.185106:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000080:00200000:2.0:1775773251.542222:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7550, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.542302:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eaaa4 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x16ca:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc1f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.542329:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x16ca:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.542383:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eaaa4 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x16ca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc1f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.542391:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.542394:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eaaa4 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x16ca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc1f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.542421:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.542431:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69f80 x1862032664029952/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.542457:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664029952:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.543325:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.548325:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664029952:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773251.548407:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x16ca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773251.548417:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x16ca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773251.548437:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x16ca:0x0](ffff9829525921c8) 00000080:00200000:2.0:1775773251.548515:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773251.548555:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5539, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773251.548561:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5539, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773251.548581:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5539,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773251.548635:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.548644:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.548654:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773251.548684:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773251.548751:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.560804:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030080:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773251.560830:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560874:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: --/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560882:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: --/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560894:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560905:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560918:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560926:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560933:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773251.560944:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b100 x1862032664030080/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773251.560971:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.560990:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4778 (0 0) 00000080:00200000:2.0:1775773251.560997:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773251.561028:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x12aa:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.561035:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x12aa:0x0], updating i_size 0 00000080:00200000:2.0:1775773251.561173:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x12aa:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773251.561184:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x12aa:0x0](ffff98296fae9988) for lock 0x12ae90f9442eaaab 00000080:00010000:2.0:1775773251.561195:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x12aa:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773251.561198:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eaaab from it ffffa414081d3df8 00010000:00010000:2.0:1775773251.561202:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.561211:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.561241:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773251.561249:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5539, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.561295:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.561317:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x12aa:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.561359:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 1/0,0 mode: --/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.561365:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.561368:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaaab lrc: 0/0,0 mode: --/PR res: [0x240000403:0x12aa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc22b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.561394:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.561401:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b100 x1862032664030336/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.561424:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030336:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.561492:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.576960:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030336:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773251.577055:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x12aa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773251.577108:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x12aa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773251.577132:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x12aa:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.577206:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773251.577224:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8669, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773251.577229:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8669, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773251.577251:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8669,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773251.577309:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.577320:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.577329:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773251.577350:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030464:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773251.577422:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.587666:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030464:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773251.587695:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587727:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587736:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587747:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587758:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587782:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587790:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587798:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773251.587810:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68380 x1862032664030464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773251.587823:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.587834:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6393 (0 0) 00000080:00200000:2.0:1775773251.587841:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773251.588119:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x18f9:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.588128:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x18f9:0x0], updating i_size 0 00000080:00200000:2.0:1775773251.588202:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x18f9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773251.588212:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x18f9:0x0](ffff98296fae9988) for lock 0x12ae90f9442eaab2 00000080:00010000:2.0:1775773251.588222:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x18f9:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773251.588225:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eaab2 from it ffffa414081d3df8 00010000:00010000:2.0:1775773251.588228:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.588250:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.588280:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773251.588288:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8669, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.588332:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.588354:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x18f9:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.588397:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.588405:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.588407:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaab2 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x18f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc263 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.588435:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.588442:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68380 x1862032664030720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.588464:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030720:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.588546:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.594676:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030720:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773251.594756:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x18f9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773251.594775:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x18f9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773251.594796:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x18f9:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.595057:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773251.595121:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6579, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773251.595126:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6579, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773251.595146:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6579,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773251.595197:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.595206:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.595216:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773251.595235:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773251.595304:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.603225:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664030848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773251.603250:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603287:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603296:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603308:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603318:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603330:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603338:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603346:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773251.603358:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b800 x1862032664030848/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773251.603371:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603383:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5348 (0 0) 00000080:00200000:2.0:1775773251.603389:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773251.603417:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x14e4:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.603425:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x14e4:0x0], updating i_size 0 00000080:00200000:2.0:1775773251.603507:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x14e4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773251.603518:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x14e4:0x0](ffff98296fae9988) for lock 0x12ae90f9442eaab9 00000080:00010000:2.0:1775773251.603642:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x14e4:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773251.603647:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eaab9 from it ffffa414081d3df8 00010000:00010000:2.0:1775773251.603651:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.603741:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.604021:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773251.604033:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6579, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.604234:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.604259:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x14e4:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.604306:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.604312:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.604315:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eaab9 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x14e4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc29b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.604340:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.604349:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b800 x1862032664031104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.604370:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664031104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.604450:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.613110:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664031104:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773251.613196:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x14e4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773251.613206:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x14e4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773251.613228:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x14e4:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.613309:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773251.613326:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4056, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773251.613331:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4056, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773251.613353:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4056,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773251.613412:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.613422:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.613431:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773251.613452:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664031232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773251.613577:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773251.633569:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664031232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773251.633594:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633624:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633633:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633643:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633666:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633680:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633688:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633695:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773251.633706:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68e00 x1862032664031232/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773251.633719:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.633729:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4087 (0 0) 00000080:00200000:2.0:1775773251.633736:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773251.634126:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xff7:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773251.634134:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xff7:0x0], updating i_size 0 00000080:00200000:2.0:1775773251.634215:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xff7:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773251.634228:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xff7:0x0](ffff98296fae9988) for lock 0x12ae90f9442eaac0 00000080:00010000:2.0:1775773251.634238:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xff7:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773251.634242:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eaac0 from it ffffa414081d3df8 00010000:00010000:2.0:1775773251.634245:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773251.634255:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.634286:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773251.634295:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4056, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773251.634339:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773251.634363:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xff7:0x0]: losing layout lock 00010000:00010000:2.0:1775773251.634405:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773251.634413:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773251.634416:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eaac0 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xff7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc2d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773251.634445:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773251.634454:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68e00 x1862032664031488/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773251.634475:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664031488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773251.634723:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773252.027507:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027518:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027538:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027551:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027565:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027572:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027580:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773252.027596:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98297c71a680 x1862032664043136/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773293 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773252.027609:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.027619:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3807 (0 0) 00000080:00200000:2.0:1775773252.027625:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773252.027790:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xedf:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773252.027798:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xedf:0x0], updating i_size 0 00000080:00200000:2.0:1775773252.028057:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xedf:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773252.028121:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xedf:0x0](ffff98296faef448) for lock 0x12ae90f9442eab99 00000080:00010000:2.0:1775773252.028133:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xedf:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773252.028136:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eab99 from it ffffa414081d3df8 00010000:00010000:2.0:1775773252.028140:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.028150:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773252.028185:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773252.028194:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3497, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773252.028241:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773252.028265:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xedf:0x0]: losing layout lock 00010000:00010000:2.0:1775773252.028309:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773252.028316:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773252.028319:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442eab99 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xedf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bc99b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773252.028340:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773252.028348:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68a80 x1862032664043392/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773252.028374:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664043392:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773252.028761:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773252.497904:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1788:0x0](ffff98296faef448) 00000080:00200000:2.0:1775773252.497912:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1788:0x0], updating i_size 0 00000080:00200000:2.0:1775773252.498007:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1788:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773252.498018:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1788:0x0](ffff98296faef448) for lock 0x12ae90f9442eacaa 00000080:00010000:2.0:1775773252.498029:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1788:0x0](ffff98296faef448) 00000080:00010000:2.0:1775773252.498033:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eacaa from it ffffa414081d3df8 00010000:00010000:2.0:1775773252.498037:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442eacaa lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1788:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bd223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.498046:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442eacaa lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1788:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bd223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773252.498106:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773252.498113:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7930, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773252.498158:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442eacaa lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1788:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bd223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773252.498184:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1788:0x0]: losing layout lock 00010000:00010000:2.0:1775773252.498232:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442eacaa lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1788:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bd223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773252.498239:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773252.498242:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7400/0x12ae90f9442eacaa lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1788:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bd223 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773252.498265:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773252.498273:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69880 x1862032664058368/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773252.498297:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664058368:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773252.498747:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000080:00200000:2.0:1775773252.758494:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x139c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773252.758503:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x139c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773252.758523:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x139c:0x0](ffff98297d9a3248) 00000080:00200000:2.0:1775773252.760011:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773252.760031:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6800, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773252.760036:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6800, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773252.760058:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6800,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773252.760137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ead3d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.760148:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ead3d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773252.760163:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ead3d lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773252.760186:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664066176:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773252.760343:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000080:00200000:2.0:1775773253.030316:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x10e7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773253.030326:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x10e7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773253.030347:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x10e7:0x0](ffff98297d9a00c8) 00000080:00200000:2.0:1775773253.030419:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773253.030437:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8599, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773253.030442:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8599, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773253.030463:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8599,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773253.030520:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eadde lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.030767:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eadde lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.030780:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eadde lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773253.030805:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664075008:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773253.031060:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773253.080337:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x165a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773253.080347:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x165a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773253.080370:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x165a:0x0](ffff98297d9a42c8) 00000080:00200000:2.0:1775773253.080446:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773253.080465:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5722, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773253.080470:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5722, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773253.080493:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5722,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773253.080582:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eadfa lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.080595:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eadfa lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.080606:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eadfa lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773253.080626:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664076544:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773253.080720:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:2.0:1775773253.187155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db19c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773253.214014:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eae40 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bded3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773253.214024:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773253.214027:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eae40 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bded3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773253.214050:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773253.214059:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6aa00 x1862032664080640/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773253.214118:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6aa00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664080640:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773253.214199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773253.295468:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664083712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773253.295571:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773253.415723:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 4/1,0 mode: --/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420087:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 4/1,0 mode: --/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420103:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420116:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420134:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420143:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420151:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773253.420167:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982945cb9c00 x1862032664085376/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773294 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773253.420179:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420191:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3518 (0 0) 00000080:00200000:2.0:1775773253.420198:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773253.420240:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xdbe:0x0](ffff98296fae9988) 00000080:00200000:2.0:1775773253.420247:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xdbe:0x0], updating i_size 0 00000080:00200000:2.0:1775773253.420330:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xdbe:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773253.420340:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xdbe:0x0](ffff98296fae9988) for lock 0x12ae90f9442eae9b 00000080:00010000:2.0:1775773253.420350:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xdbe:0x0](ffff98296fae9988) 00000080:00010000:2.0:1775773253.420354:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eae9b from it ffffa414081d3df8 00010000:00010000:2.0:1775773253.420357:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.420365:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773253.420399:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773253.420407:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3018, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773253.420453:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773253.420476:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xdbe:0x0]: losing layout lock 00010000:00010000:2.0:1775773253.420520:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 1/0,0 mode: --/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773253.420527:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773253.420530:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5000/0x12ae90f9442eae9b lrc: 0/0,0 mode: --/PR res: [0x240000403:0xdbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2be1ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773253.420553:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773253.420559:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68000 x1862032664085632/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773253.420588:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664085632:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773253.421125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773253.457492:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa6000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773253.571474:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982945cbb100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664089216:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773253.571847:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773253.606332:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7493, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773253.606338:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7493, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773253.606362:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7493,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773253.606416:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eaef6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.606426:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eaef6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.606436:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eaef6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773253.606457:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664090368:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773253.606532:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773253.783776:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773253.783796:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773253.783850:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1109:0x0](ffff98296fae5348) 00000080:00200000:2.0:1775773253.783969:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773253.783988:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7471, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773253.783993:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7471, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773253.784013:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7471,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773253.784100:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eaf4a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.784111:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eaf4a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773253.784121:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eaf4a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773253.784144:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664094976:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773253.784238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000080:00200000:2.0:1775773254.861975:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xec7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773254.861985:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xec7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773254.862008:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xec7:0x0](ffff982952597448) 00000080:00200000:2.0:1775773254.867318:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773254.867345:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f3454, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773254.867350:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f3454, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773254.867376:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f3454,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773254.867441:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb196 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773254.867452:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb196 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773254.867463:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb196 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773254.867488:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664127232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773254.867576:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00010000:00010000:2.0:1775773255.324051:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324235:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 4/1,0 mode: --/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324248:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324260:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324281:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324289:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324297:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.324316:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad4700 x1862032664137088/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.324330:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324342:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3596 (0 0) 00000080:00200000:2.0:1775773255.324349:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.324385:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xe0c:0x0](ffff98296faeec08) 00000080:00200000:2.0:1775773255.324394:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xe0c:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.324474:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xe0c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.324485:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xe0c:0x0](ffff98296faeec08) for lock 0x12ae90f9442eb23e 00000080:00010000:2.0:1775773255.324495:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xe0c:0x0](ffff98296faeec08) 00000080:00010000:2.0:1775773255.324499:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb23e from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.324502:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.324512:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.324547:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.324555:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3075, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.324614:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.324638:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xe0c:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.324685:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 1/0,0 mode: --/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.324692:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.324695:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb23e lrc: 0/0,0 mode: --/PR res: [0x200000403:0xe0c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bfec3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.324716:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.324724:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68e00 x1862032664137344/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.324824:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664137344:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.324928:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00010000:00010000:2.0:1775773255.383041:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442eb253 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2bff6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.383117:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb44:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.383168:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442eb253 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bff6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.383177:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.383180:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442eb253 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2bff6b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.383205:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.383214:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b800 x1862032664138496/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.383240:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664138496:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.383331:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773255.526166:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526178:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526190:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526203:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526224:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526233:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526241:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.526253:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296f042a00 x1862032664140928/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.526266:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526277:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2938 (0 0) 00000080:00200000:2.0:1775773255.526284:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.526326:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xb7a:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.526333:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xb7a:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.526423:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xb7a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.526435:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xb7a:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb284 00000080:00010000:2.0:1775773255.526446:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xb7a:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.526450:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb284 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.526454:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.526464:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.526496:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.526505:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1858, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.526551:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.526576:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xb7a:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.526716:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.526724:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.526727:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442eb284 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c00f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.526757:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.526764:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68380 x1862032664141184/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.526788:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141184:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.526879:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.531692:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141184:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.531789:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xb7a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.531800:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xb7a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.531821:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xb7a:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.531896:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.531924:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4546, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.531929:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4546, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.531950:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4546,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.532005:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.532016:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.532026:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.532046:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141312:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.532879:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.538377:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141312:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.538404:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538434:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538442:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538453:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538465:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538477:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538486:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538494:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.538505:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69f80 x1862032664141312/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.538519:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538529:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4332 (0 0) 00000080:00200000:2.0:1775773255.538535:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.538570:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x10ec:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.538578:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x10ec:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.538666:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x10ec:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.538677:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x10ec:0x0](ffff98296faec2c8) for lock 0x12ae90f9442eb28b 00000080:00010000:2.0:1775773255.538689:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x10ec:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773255.538692:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb28b from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.538696:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.538705:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.538734:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.538752:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4546, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.538796:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.538831:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x10ec:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.538873:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 1/0,0 mode: --/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.538881:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.538883:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb28b lrc: 0/0,0 mode: --/PR res: [0x200000403:0x10ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c012b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.538905:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.538926:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69f80 x1862032664141568/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.538949:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141568:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.539027:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.553729:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141568:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.553826:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x10ec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.553837:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x10ec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.553862:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x10ec:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.553951:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.553970:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8078, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.553974:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8078, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.553997:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8078,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.554120:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.554135:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.554144:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.554165:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141696:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.554244:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.564225:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141696:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.564253:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564286:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564294:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564305:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564316:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564329:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564341:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564347:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.564358:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6a680 x1862032664141696/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.564370:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564379:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6098 (0 0) 00000080:00200000:2.0:1775773255.564386:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.564420:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x17d2:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.564427:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x17d2:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.564511:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x17d2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.564522:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x17d2:0x0](ffff98296faec2c8) for lock 0x12ae90f9442eb292 00000080:00010000:2.0:1775773255.564532:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x17d2:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773255.564536:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb292 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.564539:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.564548:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.564577:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.564585:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8078, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.564629:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.564650:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x17d2:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.564691:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.564699:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.564702:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb292 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x17d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0163 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.564725:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.564732:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6a680 x1862032664141952/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.564755:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141952:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.564840:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.571134:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664141952:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.571227:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x17d2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.571237:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x17d2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.571261:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x17d2:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.571333:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.571350:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4542, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.571354:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4542, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.571377:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4542,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.571428:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.571442:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.571452:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.571470:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142080:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.571546:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.576566:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142080:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.576604:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576635:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576644:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576656:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576667:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576681:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576695:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576702:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.576713:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6bb80 x1862032664142080/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.576725:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576746:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4330 (0 0) 00000080:00200000:2.0:1775773255.576752:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.576789:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x10ea:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.576795:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x10ea:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.576896:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x10ea:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.576906:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x10ea:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb299 00000080:00010000:2.0:1775773255.576926:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x10ea:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.576930:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb299 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.576932:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.576941:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.576971:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.576980:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4542, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.577025:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.577045:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x10ea:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.577110:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.577117:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.577119:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb299 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x10ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c019b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.577138:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.577145:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6bb80 x1862032664142336/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.577166:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142336:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.577237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.580192:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6bb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142336:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.580286:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x10ea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.580297:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x10ea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.580319:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x10ea:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.580391:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.580406:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7055, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.580411:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7055, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.580434:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7055,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.580490:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.580504:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.580513:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.580531:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142464:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.580603:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.589144:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142464:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.589170:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589201:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589211:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589224:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589236:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589250:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589263:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589271:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.589283:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68700 x1862032664142464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.589296:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589306:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5586 (0 0) 00000080:00200000:2.0:1775773255.589313:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.589341:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x15d2:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.589348:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x15d2:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.589430:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x15d2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.589441:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x15d2:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb2a0 00000080:00010000:2.0:1775773255.589452:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x15d2:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.589455:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2a0 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.589458:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.589467:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.589498:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.589507:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f7055, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.589553:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.589575:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x15d2:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.589633:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.589641:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.589643:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a0 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x15d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c01d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.589665:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.589671:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68700 x1862032664142720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.589696:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142720:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.589776:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.611766:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142720:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.611869:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x15d2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.611880:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x15d2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.611901:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x15d2:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.611984:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.612002:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f30, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.612007:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f30, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.612029:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f30,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.612131:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.612148:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.612158:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.612179:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142848:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.612264:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.625747:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664142848:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.625776:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625817:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625826:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625836:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625847:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625860:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625874:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625881:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.625892:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6ad80 x1862032664142848/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.625904:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.625944:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2074 (0 0) 00000080:00200000:2.0:1775773255.625951:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.625989:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x81a:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.625997:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x81a:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.626133:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x81a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.626147:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x81a:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb2a7 00000080:00010000:2.0:1775773255.626158:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x81a:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.626162:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2a7 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.626165:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.626176:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.626205:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.626213:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f30, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.626259:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.626283:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x81a:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.626329:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.626337:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.626340:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2a7 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x81a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c020b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.626362:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.626369:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6ad80 x1862032664143104/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.626394:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143104:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.626472:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.634996:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6ad80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143104:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.635127:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x81a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.635139:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x81a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.635165:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x81a:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.635245:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.635263:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6066, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.635268:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6066, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.635291:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6066,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.635354:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.635369:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.635380:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.635403:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143232:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.635481:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.638427:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143232:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.638457:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638487:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: --/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638496:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: --/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638507:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638518:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638537:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638551:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638558:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.638569:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6b480 x1862032664143232/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.638581:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638604:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5042 (0 0) 00000080:00200000:2.0:1775773255.638611:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.638644:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x13b2:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.638651:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x13b2:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.638743:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x13b2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.638753:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x13b2:0x0](ffff98296faec2c8) for lock 0x12ae90f9442eb2ae 00000080:00010000:2.0:1775773255.638766:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x13b2:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773255.638769:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2ae from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.638772:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.638781:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.638823:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.638832:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6066, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.638878:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.638901:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x13b2:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.638955:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 1/0,0 mode: --/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.638962:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.638965:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2ae lrc: 0/0,0 mode: --/PR res: [0x240000403:0x13b2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0243 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.638983:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.638990:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6b480 x1862032664143488/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.639011:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143488:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.639107:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.645435:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6b480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143488:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.645519:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x13b2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.645541:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x13b2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.645564:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x13b2:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.645649:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.645665:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9598, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.645670:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9598, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.645691:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9598,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.645744:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.645759:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.645770:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.645791:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143616:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.645875:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.648520:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143616:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.648559:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648607:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648617:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648628:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648640:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648654:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648668:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648675:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.648687:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69500 x1862032664143616/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.648700:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648711:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6858 (0 0) 00000080:00200000:2.0:1775773255.648718:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.648762:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1aca:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.648769:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1aca:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.648862:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1aca:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.648872:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1aca:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb2b5 00000080:00010000:2.0:1775773255.648883:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1aca:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.648887:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2b5 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.648890:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.648898:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.648939:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.648947:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9598, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.648993:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.649017:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1aca:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.649061:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.649111:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.649115:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2b5 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1aca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c027b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.649138:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.649145:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69500 x1862032664143872/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.649168:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143872:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.649250:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.658009:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664143872:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.658138:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1aca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.658149:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1aca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.658170:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1aca:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.658253:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.658273:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f771, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.658279:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f771, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.658302:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f771,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.658363:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.658378:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.658388:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.658410:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144000:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.658482:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.669940:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144000:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.669969:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670003:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: --/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670012:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: --/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670024:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670036:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670048:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670061:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670129:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.670141:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69c00 x1862032664144000/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.670153:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670164:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2444 (0 0) 00000080:00200000:2.0:1775773255.670171:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.670199:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x98c:0x0](ffff98296fae80c8) 00000080:00200000:2.0:1775773255.670206:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x98c:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.670300:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x98c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.670312:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x98c:0x0](ffff98296fae80c8) for lock 0x12ae90f9442eb2bc 00000080:00010000:2.0:1775773255.670323:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x98c:0x0](ffff98296fae80c8) 00000080:00010000:2.0:1775773255.670327:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2bc from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.670331:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.670340:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.670369:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.670378:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f771, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.670424:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.670448:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x98c:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.670494:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 1/0,0 mode: --/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.670502:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.670505:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7400/0x12ae90f9442eb2bc lrc: 0/0,0 mode: --/PR res: [0x200000403:0x98c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c02b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.670528:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.670536:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69c00 x1862032664144256/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.670561:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144256:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.670641:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.688611:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144256:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.693473:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c35500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144384:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.693500:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693538:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693547:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693566:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693579:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693593:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693600:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693607:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.693618:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981c35500 x1862032664144384/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.693631:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693642:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/4445 (0 0) 00000080:00200000:2.0:1775773255.693648:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.693686:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x115d:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.693693:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x115d:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.693794:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x115d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.693806:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x115d:0x0](ffff98296faec2c8) for lock 0x12ae90f9442eb2c3 00000080:00010000:2.0:1775773255.693818:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x115d:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773255.693822:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2c3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.693826:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.693836:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.693870:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.693878:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4873, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.693934:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.693958:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x115d:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.694004:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.694012:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.694015:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe4400/0x12ae90f9442eb2c3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x115d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c02eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.694036:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.694043:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b68a80 x1862032664144640/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.694165:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144640:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.694238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.706053:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144640:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.706189:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x115d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.706200:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x115d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.706224:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x115d:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.706307:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.706328:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8044, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.706333:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8044, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.706356:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8044,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.706421:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.706433:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.706443:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.706468:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144768:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.706548:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.713323:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664144768:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.713364:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713405:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: --/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713414:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: --/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713432:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713443:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713456:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713465:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713471:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.713482:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b6a300 x1862032664144768/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.713494:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713504:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6031 (0 0) 00000080:00200000:2.0:1775773255.713511:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.713547:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x178f:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773255.713553:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x178f:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.713637:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x178f:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.713647:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x178f:0x0](ffff98296faec2c8) for lock 0x12ae90f9442eb2ca 00000080:00010000:2.0:1775773255.713660:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x178f:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773255.713663:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2ca from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.713667:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.713676:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.713708:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.713715:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8044, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.717003:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb2ca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x178f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0323 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.717067:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x178f:0x0]: losing layout lock 00000100:00100000:2.0:1775773255.845374:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147456:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.845476:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.852631:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981ad5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147456:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.852658:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852692:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: --/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852701:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: --/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852712:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852725:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852744:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852753:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852759:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.852770:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981ad5c00 x1862032664147456/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.852781:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852791:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5404 (0 0) 00000080:00200000:2.0:1775773255.852796:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.852825:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x151c:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.852832:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x151c:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.852933:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x151c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.852945:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x151c:0x0](ffff98296fae9148) for lock 0x12ae90f9442eb2fb 00000080:00010000:2.0:1775773255.852956:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x151c:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773255.852960:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb2fb from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.852963:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.852972:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.853009:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.853018:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6690, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.853063:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.853134:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x151c:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.853181:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 1/0,0 mode: --/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.853188:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.853191:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb2fb lrc: 0/0,0 mode: --/PR res: [0x200000403:0x151c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c04ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.853211:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.853219:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b6a300 x1862032664147712/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.853243:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.853330:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.859992:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b6a300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147712:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.860121:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x151c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.860135:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x151c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.860162:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x151c:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.860236:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.860254:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1314, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.860259:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1314, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.860282:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1314,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.860339:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.860349:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.860359:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.860377:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147840:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.860445:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.867346:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664147840:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.867378:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867409:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867417:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867428:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867438:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867451:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867458:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867464:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.867476:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b69180 x1862032664147840/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.867489:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867498:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2716 (0 0) 00000080:00200000:2.0:1775773255.867504:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.867544:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xa9c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773255.867553:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xa9c:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.867654:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xa9c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.867665:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xa9c:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb302 00000080:00010000:2.0:1775773255.867675:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xa9c:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773255.867679:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb302 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.867682:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.867690:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.867736:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.867745:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1314, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.867794:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.868161:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa9c:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.868205:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.868211:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.868214:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb302 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c04e3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.868236:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.868243:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982970b69180 x1862032664148096/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.868265:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148096:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.868350:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.872126:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b69180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148096:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.872205:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xa9c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.872215:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xa9c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.872236:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xa9c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773255.872315:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.872334:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f383, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.872339:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f383, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.872362:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f383,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.872440:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.872450:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.872459:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.872479:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148224:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.872569:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.876279:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148224:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.876305:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876337:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x898:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876346:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876358:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876368:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876380:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876388:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876396:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.876407:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92a00 x1862032664148224/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.876420:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876430:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2200 (0 0) 00000080:00200000:2.0:1775773255.876436:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.876465:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x898:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.876472:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x898:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.876561:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x898:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.876571:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x898:0x0](ffff98296fae9148) for lock 0x12ae90f9442eb309 00000080:00010000:2.0:1775773255.876594:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x898:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773255.876598:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb309 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.876601:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.876610:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.876638:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.876646:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f383, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.876693:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.876727:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x898:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.876769:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.876776:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.876779:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb309 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x898:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c051b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.876810:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.876818:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92a00 x1862032664148480/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.876838:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148480:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.876915:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.884544:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148480:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.884629:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x898:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.884638:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x898:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.884658:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x898:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.884744:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.884759:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f5350, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.884764:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f5350, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.884783:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f5350,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.884847:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.884861:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.884871:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.884891:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148608:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.884971:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.895325:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148608:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.895352:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895384:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895392:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895403:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895415:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895429:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895443:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895450:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.895462:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90e00 x1862032664148608/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.895477:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895487:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4734 (0 0) 00000080:00200000:2.0:1775773255.895493:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.895529:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x127e:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.895537:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x127e:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.895630:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x127e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.895640:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x127e:0x0](ffff98296fae9148) for lock 0x12ae90f9442eb310 00000080:00010000:2.0:1775773255.895650:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x127e:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773255.895653:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb310 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.895656:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.895664:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.895705:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773255.895721:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f5350, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773255.895764:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.895786:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x127e:0x0]: losing layout lock 00010000:00010000:2.0:1775773255.896123:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773255.896133:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773255.896135:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6c00/0x12ae90f9442eb310 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x127e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0553 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773255.896156:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773255.896164:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90e00 x1862032664148864/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773255.896184:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148864:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773255.896259:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.901996:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148864:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773255.902144:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x127e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773255.902155:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x127e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773255.902179:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x127e:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.902259:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773255.902277:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6022, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773255.902283:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6022, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773255.902307:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6022,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773255.902365:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.902381:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.902391:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773255.902414:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148992:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773255.902500:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773255.912427:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664148992:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773255.912456:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912489:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912497:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912508:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912517:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912541:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912553:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912560:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773255.912572:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90a80 x1862032664148992/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773296 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773255.912595:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912605:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5020 (0 0) 00000080:00200000:2.0:1775773255.912613:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773255.912644:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x139c:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773255.912651:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x139c:0x0], updating i_size 0 00000080:00200000:2.0:1775773255.912737:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x139c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773255.912748:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x139c:0x0](ffff98296fae9148) for lock 0x12ae90f9442eb317 00000080:00010000:2.0:1775773255.912760:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x139c:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773255.912764:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb317 from it ffffa414081d3df8 00010000:00010000:2.0:1775773255.912768:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773255.912777:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6000/0x12ae90f9442eb317 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x139c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c058b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773255.913226:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.201349:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xf61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.201360:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xf61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.201384:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xf61:0x0](ffff98297d9a0908) 00000080:00200000:2.0:1775773256.201455:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.201471:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8067, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.201476:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8067, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.201496:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8067,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.201560:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.201570:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.202809:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.202840:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155136:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.202932:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.206832:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155136:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773256.206862:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206892:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206911:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206928:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206949:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206962:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206972:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.206979:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.206992:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90a80 x1862032664155136/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.207006:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.207017:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6092 (0 0) 00000080:00200000:2.0:1775773256.207023:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.207061:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x17cc:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.207107:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x17cc:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.207188:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x17cc:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.207198:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x17cc:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb387 00000080:00010000:2.0:1775773256.207209:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x17cc:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773256.207213:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb387 from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.207216:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.207225:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.207258:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.207268:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8067, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.207313:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.207336:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x17cc:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.207386:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.207392:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.207395:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb387 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x17cc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c090b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.207416:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.207423:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90a80 x1862032664155392/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.207450:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155392:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.207539:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.215524:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155392:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773256.215633:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x17cc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.215643:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x17cc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.215665:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x17cc:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.215740:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.215758:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f4627, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.215763:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f4627, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.215798:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f4627,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.215855:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.215866:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.215876:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.215896:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155520:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.216000:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.227587:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155520:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773256.227630:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227678:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227688:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227705:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227720:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227733:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227749:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227759:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.227775:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91500 x1862032664155520/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.227805:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.227820:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/4372 (0 0) 00000080:00200000:2.0:1775773256.227826:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.227869:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1114:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.227876:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1114:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.227981:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1114:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.227993:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1114:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb38e 00000080:00010000:2.0:1775773256.228005:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1114:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773256.228009:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb38e from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.228012:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.228022:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.228056:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.228107:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f4627, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.228156:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.228181:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1114:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.228228:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.228235:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.228237:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb38e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1114:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0943 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.228257:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.228264:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91500 x1862032664155776/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.228287:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155776:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.228359:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.240131:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155776:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773256.240212:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1114:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.240223:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1114:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.240246:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1114:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.240318:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.240341:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2730, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.240346:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2730, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.240370:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2730,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.240430:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.240440:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.240450:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.240473:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155904:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.240544:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.248155:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664155904:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773256.248180:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248213:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248223:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248241:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248254:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248267:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248275:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248282:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.248293:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90000 x1862032664155904/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.248306:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248316:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3374 (0 0) 00000080:00200000:2.0:1775773256.248322:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.248355:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xd2e:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.248362:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xd2e:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.248448:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xd2e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.248459:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xd2e:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb395 00000080:00010000:2.0:1775773256.248471:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xd2e:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773256.248475:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb395 from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.248478:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.248488:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.248523:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.248532:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2730, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.248597:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.248622:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xd2e:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.248667:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.248674:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.248677:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb395 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xd2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c097b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.248698:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.248705:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90000 x1862032664156160/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.248728:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156160:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.248805:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.256738:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156160:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773256.256837:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xd2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.256847:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xd2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.256866:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xd2e:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.256961:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.256979:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1105, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.256984:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1105, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.257006:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1105,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.257107:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.257117:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.257128:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.257151:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156288:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.257227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.264290:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156288:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773256.264314:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264341:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264347:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: --/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264360:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264367:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264376:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264381:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264386:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.264392:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90700 x1862032664156288/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.264400:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264408:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2611 (0 0) 00000080:00200000:2.0:1775773256.264413:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.264451:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xa33:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.264457:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xa33:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.264529:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xa33:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.264539:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xa33:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb39c 00000080:00010000:2.0:1775773256.264549:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xa33:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773256.264553:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb39c from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.264556:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.264566:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.264598:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.264607:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1105, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.264654:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.264685:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xa33:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.264729:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 1/0,0 mode: --/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.264737:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.264740:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb39c lrc: 0/0,0 mode: --/PR res: [0x200000403:0xa33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c09b3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.264762:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.264769:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90700 x1862032664156544/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.264791:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156544:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.264867:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.275607:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156544:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773256.275704:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xa33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.275715:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xa33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.275738:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xa33:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.275826:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.275844:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.275848:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.275870:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6486,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.275934:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.276198:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.276210:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.276232:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156672:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.276310:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.280515:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156672:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773256.280723:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280755:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280764:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280781:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280872:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280886:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280894:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.280912:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.280924:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91180 x1862032664156672/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.280936:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.281740:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/5252 (0 0) 00000080:00200000:2.0:1775773256.281746:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.281777:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1484:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773256.281784:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1484:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.282169:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1484:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.282182:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1484:0x0](ffff98296faeba88) for lock 0x12ae90f9442eb3a3 00000080:00010000:2.0:1775773256.282195:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1484:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773256.282199:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb3a3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.282203:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.282214:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.282255:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.282264:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.282315:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.282341:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1484:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.282390:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.282398:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.282401:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb3a3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1484:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c09eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.282428:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.282443:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91180 x1862032664156928/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.282473:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664156928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.635676:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773256.643237:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664164224:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773256.643346:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x991:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773256.643356:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x991:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773256.643379:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x991:0x0](ffff98297d9a6c08) 00000080:00200000:2.0:1775773256.643467:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773256.643485:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8114, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773256.643491:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8114, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773256.643512:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8114,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773256.644255:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb42f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.644267:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb42f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.644279:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb42f lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773256.644303:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664164352:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773256.644474:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00010000:00010000:2.0:1775773256.799250:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799270:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799283:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799296:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799316:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799326:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799334:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773256.799351:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965189180 x1862032664166656/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773297 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773256.799364:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799375:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5434 (0 0) 00000080:00200000:2.0:1775773256.799383:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773256.799423:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x153a:0x0](ffff98296faee3c8) 00000080:00200000:2.0:1775773256.799430:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x153a:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.799516:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x153a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.799527:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x153a:0x0](ffff98296faee3c8) for lock 0x12ae90f9442eb459 00000080:00010000:2.0:1775773256.799540:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x153a:0x0](ffff98296faee3c8) 00000080:00010000:2.0:1775773256.799544:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb459 from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.799547:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.799556:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.799695:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.799703:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6751, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.799756:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.799780:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x153a:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.800022:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.800029:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.800032:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb459 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x153a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c0f9b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.800062:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.800191:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91f80 x1862032664166912/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.800221:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664166912:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.800354:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15de40 going to sleep for 36 seconds 00000080:00200000:2.0:1775773256.867493:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xdf9:0x0](ffff98296faecb08) 00000080:00200000:2.0:1775773256.867502:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xdf9:0x0], updating i_size 0 00000080:00200000:2.0:1775773256.867611:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xdf9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773256.867622:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xdf9:0x0](ffff98296faecb08) for lock 0x12ae90f9442eb483 00000080:00010000:2.0:1775773256.867633:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xdf9:0x0](ffff98296faecb08) 00000080:00010000:2.0:1775773256.867637:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb483 from it ffffa414081d3df8 00010000:00010000:2.0:1775773256.867640:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb483 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xdf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c10eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773256.867654:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb483 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xdf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c10eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.867688:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773256.867696:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3037, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773256.867737:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb483 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xdf9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c10eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773256.867759:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xdf9:0x0]: losing layout lock 00010000:00010000:2.0:1775773256.867817:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb483 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xdf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c10eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773256.867825:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773256.867827:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb483 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xdf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c10eb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773256.867848:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773256.867857:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93b80 x1862032664169216/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773256.867880:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664169216:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773256.867980:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dc00 going to sleep for 36 seconds 00010000:00010000:2.0:1775773257.039716:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039727:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039743:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039754:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039768:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039776:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039795:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.039807:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9e680 x1862032664174720/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.039819:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.039830:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6884 (0 0) 00000080:00200000:2.0:1775773257.039838:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.039873:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1ae4:0x0](ffff98296fae8908) 00000080:00200000:2.0:1775773257.039880:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1ae4:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.039984:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1ae4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.039996:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1ae4:0x0](ffff98296fae8908) for lock 0x12ae90f9442eb4ec 00000080:00010000:2.0:1775773257.040007:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1ae4:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773257.040011:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb4ec from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.040014:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.040024:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.040055:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.040090:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9751, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.040141:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.040164:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1ae4:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.040219:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.040228:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.040231:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442eb4ec lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1ae4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1433 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.040254:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.040262:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93100 x1862032664174976/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.040286:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664174976:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.040772:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d000 going to sleep for 36 seconds 00010000:00010000:2.0:1775773257.289173:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442eb57f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1271:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c18cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.289204:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1271:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.289249:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442eb57f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1271:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c18cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.289256:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.289258:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442eb57f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1271:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c18cb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.289280:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.289288:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93480 x1862032664183040/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.289314:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664183040:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.289442:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d840 going to sleep for 36 seconds 00000080:00200000:2.0:1775773257.353014:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xa71:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.353025:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xa71:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.353048:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xa71:0x0](ffff9829525921c8) 00000080:00200000:2.0:1775773257.353172:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.353198:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6872, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.353204:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6872, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.353229:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6872,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.353288:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442eb5a2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.353300:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442eb5a2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.353315:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442eb5a2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.353338:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664184704:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.353486:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15df00 going to sleep for 36 seconds 00000080:00200000:2.0:1775773257.508303:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1474:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.508313:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1474:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.508339:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1474:0x0](ffff98297d9a1148) 00000080:00200000:2.0:1775773257.508427:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.508446:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8844, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.508452:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8844, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.508473:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8844,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.508627:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eb5e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.508638:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eb5e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.508651:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6200/0x12ae90f9442eb5e8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.508673:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664188544:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.508797:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d300 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.533632:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff98294f4e5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664189568:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.533763:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dd80 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.611162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa63c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.614356:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c34380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192256:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773257.614441:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xa9f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.614452:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xa9f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.614474:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xa9f:0x0](ffff98296fae3248) 00000080:00200000:2.0:1775773257.614741:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.614759:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8468, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.614763:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8468, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.614854:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8468,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.614916:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.614927:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.614938:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.614958:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192384:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.615047:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.617693:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192384:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773257.617721:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617754:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617764:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617849:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617858:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617869:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617886:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617892:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.617903:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90380 x1862032664192384/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.617915:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.617926:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6243 (0 0) 00000080:00200000:2.0:1775773257.617933:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.617961:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1863:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773257.617967:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1863:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.618031:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1863:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.618038:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1863:0x0](ffff98296faedb88) for lock 0x12ae90f9442eb62e 00000080:00010000:2.0:1775773257.618047:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1863:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773257.618049:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb62e from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.618052:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.618058:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.618232:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.618239:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8468, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.618278:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.618293:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1863:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.618329:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.618333:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.618335:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb62e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1863:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1e43 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.618349:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.618353:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90380 x1862032664192640/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.618369:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192640:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.618458:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.621969:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192640:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773257.622053:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x1863:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.622061:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x1863:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.622204:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1863:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773257.622272:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.622287:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f2323, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.622290:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f2323, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.622311:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2323,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.622359:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.622367:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.622375:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.622391:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192768:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.622487:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.627462:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664192768:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773257.627491:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627524:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627544:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627567:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627578:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627590:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627598:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627606:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.627616:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91880 x1862032664192768/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.627629:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.627637:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3170 (0 0) 00000080:00200000:2.0:1775773257.627643:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.627675:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xc62:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773257.627681:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xc62:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.627762:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xc62:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.628013:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xc62:0x0](ffff98296faeaa08) for lock 0x12ae90f9442eb635 00000080:00010000:2.0:1775773257.628027:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xc62:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773257.628031:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb635 from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.628034:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.628043:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.628094:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.628103:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2323, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.628151:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.628175:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc62:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.628222:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.628230:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.628233:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb635 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1e7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.628254:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.628262:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91880 x1862032664193024/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.628284:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193024:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.628420:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.633330:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193024:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773257.633422:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xc62:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.633433:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xc62:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.633455:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xc62:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773257.633522:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.633747:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8604, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.633752:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8604, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.633865:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8604,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.633933:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.633943:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.633953:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.633974:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193152:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.634265:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.637895:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193152:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773257.637921:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.637956:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.637966:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: --/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.637983:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.637994:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.638007:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.638016:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.638024:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.638035:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92300 x1862032664193152/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.638048:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.638058:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6311 (0 0) 00000080:00200000:2.0:1775773257.638237:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.638273:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x18a7:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773257.638280:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x18a7:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.638363:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x18a7:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.638373:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x18a7:0x0](ffff98296faedb88) for lock 0x12ae90f9442eb63c 00000080:00010000:2.0:1775773257.638385:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x18a7:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773257.638388:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb63c from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.638391:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.638401:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.638427:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.638435:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8604, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.638478:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.638501:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x18a7:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.638758:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.638766:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.638864:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb63c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x18a7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1eb3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.638900:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.638909:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92300 x1862032664193408/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.638934:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193408:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.639087:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.643293:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193408:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773257.643377:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0x18a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.643387:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0x18a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.643408:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0x18a7:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773257.643482:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.643498:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.643503:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.643523:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9486,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.643965:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.643976:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.643986:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.644007:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193536:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.644148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.647587:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193536:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773257.647613:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647641:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647650:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647666:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647676:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647687:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647695:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647702:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.647713:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f93800 x1862032664193536/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.647724:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647734:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6802 (0 0) 00000080:00200000:2.0:1775773257.647741:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.647771:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1a92:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773257.647777:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1a92:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.647854:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1a92:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.647864:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1a92:0x0](ffff98296faeaa08) for lock 0x12ae90f9442eb643 00000080:00010000:2.0:1775773257.647893:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1a92:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773257.647897:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb643 from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.647900:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.647909:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.647937:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.647945:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9486, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.647988:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.648008:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1a92:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.648051:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.648058:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.648061:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6e00/0x12ae90f9442eb643 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1a92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1eeb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.648295:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.648302:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93800 x1862032664193792/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.648324:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193792:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.648397:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.652147:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f93800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193792:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773257.652233:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1a92:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.652243:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1a92:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.652265:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1a92:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773257.652336:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.652351:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1965, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.652355:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1965, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.652375:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1965,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.652430:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.652440:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.652450:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.652470:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193920:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.652538:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773257.656297:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664193920:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773257.656326:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656359:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: --/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656368:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: --/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656386:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656400:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656414:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656423:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656430:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773257.656441:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92d80 x1862032664193920/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773298 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773257.656453:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656465:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3041 (0 0) 00000080:00200000:2.0:1775773257.656472:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773257.656507:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xbe1:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773257.656514:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xbe1:0x0], updating i_size 0 00000080:00200000:2.0:1775773257.656617:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xbe1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773257.656628:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xbe1:0x0](ffff98296faedb88) for lock 0x12ae90f9442eb64a 00000080:00010000:2.0:1775773257.656638:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xbe1:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773257.656641:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb64a from it ffffa414081d3df8 00010000:00010000:2.0:1775773257.656644:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.656653:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.656684:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773257.656694:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1965, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773257.656742:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773257.656769:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xbe1:0x0]: losing layout lock 00010000:00010000:2.0:1775773257.656821:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 1/0,0 mode: --/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.656829:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.656831:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb64a lrc: 0/0,0 mode: --/PR res: [0x200000403:0xbe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c1f23 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.656853:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.656860:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92d80 x1862032664194176/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.656897:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664194176:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.656977:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00010000:00010000:2.0:1775773257.708099:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442eb666 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x110c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2003 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773257.708114:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773257.708116:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442eb666 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x110c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2003 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773257.708141:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773257.708152:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91c00 x1862032664195712/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773257.708178:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664195712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773257.708297:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000080:00200000:2.0:1775773257.865513:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xd08:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773257.865523:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xd08:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773257.865546:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xd08:0x0](ffff982952590908) 00000080:00200000:2.0:1775773257.865624:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773257.865646:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f471, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773257.865651:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f471, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773257.865672:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f471,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773257.865737:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442eb6a5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.865747:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442eb6a5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773257.865757:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442eb6a5 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773257.865776:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664198912:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773257.865852:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.019127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1000 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.022748:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff98296518d880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664203904:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.023020:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023056:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023222:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023237:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023250:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023265:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023274:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023282:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.023295:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff98296518d880 x1862032664203904/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.023308:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023318:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/3306 (0 0) 00000080:00200000:2.0:1775773258.023325:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.023353:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xcea:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773258.023360:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xcea:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.023444:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xcea:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.023456:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xcea:0x0](ffff98296faeaa08) for lock 0x12ae90f9442eb700 00000080:00010000:2.0:1775773258.023468:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xcea:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773258.023471:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb700 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.023475:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.023484:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.023514:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.023524:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f2494, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.023582:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.023606:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xcea:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.023653:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.023661:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.023664:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166aa00/0x12ae90f9442eb700 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xcea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c24d3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.023686:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.023692:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91f80 x1862032664204160/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.023718:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204160:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.023804:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.029507:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204160:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773258.029623:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xcea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.029636:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xcea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.029659:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xcea:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773258.029736:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.029755:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1700, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.030046:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1700, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.030116:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1700,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.030205:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.030218:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.030230:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.030255:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204288:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.030448:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.036947:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204288:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.036980:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037016:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037025:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037037:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037049:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037062:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037122:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037129:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.037139:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90700 x1862032664204288/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.037151:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037162:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2859 (0 0) 00000080:00200000:2.0:1775773258.037169:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.037202:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xb2b:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.037209:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xb2b:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.037300:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xb2b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.037310:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xb2b:0x0](ffff98296faea1c8) for lock 0x12ae90f9442eb707 00000080:00010000:2.0:1775773258.037320:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xb2b:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773258.037324:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb707 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.037327:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.037335:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.037365:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.037373:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1700, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.037419:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.037443:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xb2b:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.037492:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.037499:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.037502:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb707 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xb2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c250b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.037529:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.037536:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90700 x1862032664204544/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.037646:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204544:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.037727:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.043297:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204544:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773258.043376:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000403:0xb2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.043385:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000403:0xb2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.043405:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000403:0xb2b:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.043479:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.043495:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f8893, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.043500:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f8893, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.043521:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f8893,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.043602:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.043612:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.043622:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.043643:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204672:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.043715:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.048953:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204672:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.048980:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049010:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049019:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049031:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049042:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049055:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049109:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049119:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.049132:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f91500 x1862032664204672/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.049144:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049155:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/6455 (0 0) 00000080:00200000:2.0:1775773258.049162:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.049192:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1937:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773258.049200:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x1937:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.049277:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1937:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.049289:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1937:0x0](ffff98296faeaa08) for lock 0x12ae90f9442eb70e 00000080:00010000:2.0:1775773258.049299:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1937:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773258.049302:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb70e from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.049305:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.049315:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.049344:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.049352:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8893, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.049397:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.049421:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1937:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.049462:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.049469:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.049472:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442eb70e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1937:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2543 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.049498:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.049504:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91500 x1862032664204928/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.049527:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.049618:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.053112:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f91500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664204928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.059913:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c35c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664205056:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.059943:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.059981:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.059990:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: --/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060002:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060013:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060026:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060039:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060047:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.060057:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981c35c00 x1862032664205056/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.060103:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060113:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/3782 (0 0) 00000080:00200000:2.0:1775773258.060121:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.060158:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0xec6:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.060166:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0xec6:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.060257:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0xec6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.060268:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0xec6:0x0](ffff98296faea1c8) for lock 0x12ae90f9442eb715 00000080:00010000:2.0:1775773258.060278:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0xec6:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773258.060281:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb715 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.060284:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.060293:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.060326:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.060334:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f3546, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.060374:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.060394:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xec6:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.060432:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.060438:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.060441:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5800/0x12ae90f9442eb715 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xec6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c257b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.060461:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.060468:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90a80 x1862032664205312/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.060489:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664205312:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773258.181380:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6843, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.181453:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442eb74d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1568:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c273b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.181479:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1568:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.181548:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442eb74d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1568:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c273b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.181555:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.181557:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5600/0x12ae90f9442eb74d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1568:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c273b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.181581:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.181591:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90e00 x1862032664208384/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.181617:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664208384:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.181702:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.224438:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xccb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.224450:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xccb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.224474:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xccb:0x0](ffff9829525900c8) 00000080:00200000:2.0:1775773258.224946:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.224967:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f1624, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.224972:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f1624, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.224995:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1624,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.225055:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.225206:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.225220:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.225249:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664209664:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.225343:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.229772:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664209664:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.229799:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229831:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229840:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: --/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229853:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229878:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229892:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229900:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229906:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.229915:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f92680 x1862032664209664/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.229927:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.229936:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/2871 (0 0) 00000080:00200000:2.0:1775773258.229942:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.229978:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0xb37:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.229984:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0xb37:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.230293:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0xb37:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.230309:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0xb37:0x0](ffff98296faea1c8) for lock 0x12ae90f9442eb769 00000080:00010000:2.0:1775773258.230319:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0xb37:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773258.230322:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb769 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.230326:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.230335:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.230364:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.230371:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f1624, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.230416:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.230438:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0xb37:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.230478:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 1/0,0 mode: --/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.230485:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.230488:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442eb769 lrc: 0/0,0 mode: --/PR res: [0x200000403:0xb37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c281b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.230507:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.230513:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92680 x1862032664209920/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.230989:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664209920:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.231126:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.236455:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f92680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664209920:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773258.236845:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0xb37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.236935:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0xb37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.236965:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0xb37:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.237045:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.237063:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f9874, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.237170:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f9874, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.237193:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f9874,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.237263:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb770 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.237275:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb770 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.237288:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442eb770 lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.237314:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664210048:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.237460:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15dcc0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.441237:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.441255:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f6937, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.441259:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f6937, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.441280:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f6937,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.441332:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.441341:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.441351:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.441369:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664217344:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.441472:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.448487:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664217344:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773258.448517:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448695:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448703:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448714:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448726:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448740:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448748:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448839:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773258.448852:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982947f90000 x1862032664217344/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773299 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773258.448875:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.448885:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/5527 (0 0) 00000080:00200000:2.0:1775773258.448892:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773258.448929:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1597:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.448937:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1597:0x0], updating i_size 0 00000080:00200000:2.0:1775773258.449035:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1597:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.449047:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1597:0x0](ffff98296faea1c8) for lock 0x12ae90f9442eb7f5 00000080:00010000:2.0:1775773258.449059:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1597:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773258.449186:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb7f5 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.449192:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.449200:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.449238:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.449246:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f6937, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.449293:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.449321:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1597:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.449366:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.449373:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.449375:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442eb7f5 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1597:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c2c7b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.449398:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.449406:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f90000 x1862032664217600/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.449434:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664217600:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.449638:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000100:00100000:2.0:1775773258.453950:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982947f90000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664217600:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773258.454035:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x1597:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.454045:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x1597:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.454094:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1597:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773258.454168:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.454184:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7821, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.454189:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7821, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.454210:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7821,[0x0:0x0:0x0]) in obj [0x240000400:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.454263:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb7fc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.454273:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb7fc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.454284:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442eb7fc lrc: 3/1,0 mode: --/CR res: [0x240000400:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.454304:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664217728:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.454421:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d780 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.752330:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0xc44:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.752392:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb8a4 lrc: 1/0,0 mode: --/PR res: [0x240000403:0xc44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c31f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.752399:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.752401:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442eb8a4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0xc44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c31f3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.752423:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.752430:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f92a00 x1862032664227200/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.752454:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f92a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664227200:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.752897:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.794447:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x200000403:0x10c6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775773258.794458:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x200000403:0x10c6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773258.794492:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000403:0x10c6:0x0](ffff9829525921c8) 00000080:00200000:2.0:1775773258.794648:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773258.794666:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f7613, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), flags=0 00000080:00200000:2.0:1775773258.794670:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f7613, dir=[0x240000402:0x1b:0x0](ffff9829525963c8), intent=getattr 00000002:00010000:2.0:1775773258.794691:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f7613,[0x0:0x0:0x0]) in obj [0x200000401:0xa:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773258.794751:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442eb8b9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.794762:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442eb8b9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.794772:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442eb8b9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773258.794792:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664228096:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773258.795546:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15da80 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.857414:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x1a11:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773258.857426:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1a11:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442eb8d5 00000080:00010000:2.0:1775773258.857436:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1a11:0x0](ffff98297d9a3a88) 00000080:00010000:2.0:1775773258.857440:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb8d5 from it ffffa414081d3df8 00010000:00010000:2.0:1775773258.857443:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442eb8d5 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1a11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c337b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773258.857453:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442eb8d5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1a11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c337b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.857504:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773258.857512:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f9328, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.857574:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442eb8d5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1a11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c337b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.857600:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1a11:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.858108:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442eb8d5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1a11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c337b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.858117:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.858121:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442eb8d5 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1a11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c337b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.858147:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.858155:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f93b80 x1862032664229888/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.858181:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f93b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664229888:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.858272:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000080:00200000:2.0:1775773258.918904:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f637, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773258.919001:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb8f1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x917:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c345b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773258.919029:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x917:0x0]: losing layout lock 00010000:00010000:2.0:1775773258.919110:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb8f1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x917:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c345b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773258.919118:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773258.919120:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb8f1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x917:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c345b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773258.919145:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773258.919157:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982947f91180 x1862032664231424/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773258.919182:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982947f91180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664231424:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773258.919326:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00010000:00010000:2.0:1775773259.026455:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x856:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026483:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026496:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026509:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026523:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026541:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026548:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.026562:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982974b2ce00 x1862032664234240/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.026575:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026586:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677443/2134 (0 0) 00000080:00200000:2.0:1775773259.026593:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773259.026639:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x856:0x0](ffff98296faea1c8) 00000080:00200000:2.0:1775773259.026646:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x856:0x0], updating i_size 0 00000080:00200000:2.0:1775773259.026729:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x856:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773259.026740:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x856:0x0](ffff98296faea1c8) for lock 0x12ae90f9442eb929 00000080:00010000:2.0:1775773259.026751:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x856:0x0](ffff98296faea1c8) 00000080:00010000:2.0:1775773259.026755:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb929 from it ffffa414081d3df8 00010000:00010000:2.0:1775773259.026758:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.026768:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.026799:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773259.026807:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f251, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773259.026864:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.026891:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x856:0x0]: losing layout lock 00010000:00010000:2.0:1775773259.026953:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.026961:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773259.026964:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7e00/0x12ae90f9442eb929 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x856:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c361b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773259.026984:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773259.026991:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4a80 x1862032664234496/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773259.027017:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664234496:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773259.027133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00000080:00200000:2.0:1775773259.066381:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000403:0x177d:0x0](ffff98296faeaa08) 00000080:00200000:2.0:1775773259.066390:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000403:0x177d:0x0], updating i_size 0 00000080:00200000:2.0:1775773259.066475:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000403:0x177d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773259.066486:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000403:0x177d:0x0](ffff98296faeaa08) for lock 0x12ae90f9442eb945 00000080:00010000:2.0:1775773259.066497:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x177d:0x0](ffff98296faeaa08) 00000080:00010000:2.0:1775773259.066501:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb945 from it ffffa414081d3df8 00010000:00010000:2.0:1775773259.066505:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442eb945 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x177d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c36fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.066515:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442eb945 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x177d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c36fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.066614:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773259.066623:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8008, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773259.066675:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442eb945 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x177d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c36fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.066699:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x177d:0x0]: losing layout lock 00010000:00010000:2.0:1775773259.066743:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442eb945 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x177d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c36fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.066750:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773259.066753:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442eb945 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x177d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c36fb expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773259.066778:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773259.066787:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6300 x1862032664236032/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773259.066904:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664236032:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773259.067142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d540 going to sleep for 36 seconds 00010000:00010000:2.0:1775773259.123584:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.123595:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.123738:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.124373:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.124390:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.124399:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.124407:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.124418:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982980f9dc00 x1862032664237312/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/584 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.124430:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.124441:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935619/6194 (0 0) 00000080:00200000:2.0:1775773259.124447:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0xa:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773259.124483:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1832:0x0](ffff98296faedb88) 00000080:00200000:2.0:1775773259.124490:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000403:0x1832:0x0], updating i_size 0 00000080:00200000:2.0:1775773259.125368:0:412860:0:(file.c:6766:ll_layout_conf()) [0x200000403:0x1832:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1775773259.125385:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1832:0x0](ffff98296faedb88) for lock 0x12ae90f9442eb961 00000080:00010000:2.0:1775773259.125396:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1832:0x0](ffff98296faedb88) 00000080:00010000:2.0:1775773259.125399:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442eb961 from it ffffa414081d3df8 00010000:00010000:2.0:1775773259.125403:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.125413:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.125445:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff9829525963c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773259.125452:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f8270, dir=[0x240000402:0x1b:0x0](ffff9829525963c8) 00010000:00010000:2.0:1775773259.125497:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.125520:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1832:0x0]: losing layout lock 00010000:00010000:2.0:1775773259.126268:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.126277:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773259.126360:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442eb961 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1832:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c37db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773259.126395:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773259.126405:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6680 x1862032664237568/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773259.126429:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664237568:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773259.126567:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d240 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.692705:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982974fa6480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.817877:0:404324:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773259.910983:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664263552:192.168.201.118@tcp:37:rm.0 00010000:00010000:2.0:1775773259.911599:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebb2f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x451:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c460c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773259.911615:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebb2f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x451:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c460c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1775773259.911662:0:412860:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x200000402:0x451:0x0](ffff982952595b88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1775773259.911679:0:412860:0:(file.c:447:ll_file_release()) START file d43j.sanityn:[0x200000402:0x451:0x0](ffff982952595b88), flags 704000 00000002:00100000:2.0:1775773259.911748:0:412860:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff982971f95500 x1862032664263424/t12884953464(12884953464) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/664 e 0 to 0 dl 1775773300 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775773259.911759:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775773259.911764:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4380 x1862032664263808/t0(0) o35->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:23/10 lens 392/840 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775773259.911786:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664263808:192.168.201.118@tcp:35:rm.0 00000100:00100000:2.0:1775773259.911913:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775773259.914116:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664263808:192.168.201.118@tcp:35:rm.0 00000002:00100000:2.0:1775773259.914149:0:412860:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff982971f95500 x1862032664263424/t12884953464(12884953464) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/664 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775773259.914219:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988),name=/ 00000002:00010000:2.0:1775773259.914233:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.914248:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.914260:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775773259.914268:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775773259.914273:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) 00000080:00010000:2.0:1775773259.914277:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) for lock 0x12ae90f9442ebb28 00000080:00010000:2.0:1775773259.914282:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb28 from it ffffa414081d3c10 00010000:00010000:2.0:1775773259.914286:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.914294:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775773259.914307:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988), inode mode 41ff mask 1 00000080:00200000:2.0:1775773259.914317:0:412860:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d43j.sanityn, flags=0 00000080:00200000:2.0:1775773259.914323:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988),name=/ 00000002:00010000:2.0:1775773259.914328:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.914333:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773259.914342:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775773259.914346:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775773259.914349:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) 00000080:00010000:2.0:1775773259.914350:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) for lock 0x12ae90f9442ebb28 00000080:00010000:2.0:1775773259.914353:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb28 from it ffffa414081d3d88 00010000:00010000:2.0:1775773259.914355:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773259.914359:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1775773259.914363:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988), inode mode 41ff mask 3 00000080:00200000:2.0:1775773259.914367:0:412860:0:(namei.c:2339:ll_rmdir()) VFS Op:name=d43j.sanityn, dir=[0x200000007:0x1:0x0](ffff982949999988) 00010000:00010000:2.0:1775773259.914381:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773259.914401:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebb2f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x451:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c460c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773259.914450:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebb2f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x451:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c460c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773259.914455:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773259.914461:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1775773259.914463:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebb2f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x451:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c460c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773259.914474:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebb28 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c45f0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1775773259.914485:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773259.914489:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4380 x1862032664264064/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 496/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773259.914503:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264064:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773259.914573:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.919524:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264064:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773259.919704:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200000402:0x451:0x0](ffff982952595b88) 00000080:00200000:2.0:1775773259.919861:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988),name=/ 00000002:00010000:2.0:1775773259.919884:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.919939:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.919949:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.919958:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773259.919977:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264192:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773259.920156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.923474:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264192:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773259.923499:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923512:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923524:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923545:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923557:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923565:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923573:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.923584:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032664264192/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/688 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.923596:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923612:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1775773259.923618:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1775773259.923644:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1775773259.923650:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) for lock 0x12ae90f9442ebb3d 00000080:00010000:2.0:1775773259.923655:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) 00000080:00010000:2.0:1775773259.923658:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb3d from it ffffa414081d3b58 00010000:00010000:2.0:1775773259.923662:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923671:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.923694:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988), inode mode 41ff mask 1 00000080:00200000:2.0:1775773259.923725:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=d60.sanityn, dir=[0x200000007:0x1:0x0](ffff982949999988), flags=8 00000080:00200000:2.0:1775773259.923742:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=d60.sanityn, dir=[0x200000007:0x1:0x0](ffff982949999988), intent=getattr 00000002:00010000:2.0:1775773259.923763:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d60.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.923800:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923807:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.923815:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773259.923843:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264320:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773259.923914:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.932343:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264320:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773259.932374:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932399:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932408:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932419:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932430:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932442:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932449:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932455:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.932466:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032664264320/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 584/672 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.932478:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932488:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589935618/1835 (0 0) 00000080:00200000:2.0:1775773259.932493:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773259.932628:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200000402:0x72b:0x0](ffff98296fae8908) 00000002:00010000:2.0:1775773259.932661:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd0:0x6d:0x0]) in obj [0x200000bd0:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.932713:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932721:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.932741:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773259.932761:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264448:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773259.932882:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.937710:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264448:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773259.937733:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937747:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 4/1,0 mode: --/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937761:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 4/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937775:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 4/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937794:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 5/2,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937802:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 5/2,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937809:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 4/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.937820:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5f80 x1862032664264448/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.937843:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937856:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589937616/109 (0 0) 00010000:00010000:2.0:1775773259.937864:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937872:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1775773259.937899:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000404:0x6d:0x0]) in obj [0x240000404:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.937945:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x6d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937953:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.937962:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773259.937983:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264576:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773259.938053:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773259.946198:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264576:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773259.946223:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x6d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946235:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 4/1,0 mode: --/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946246:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946258:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946347:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946356:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946364:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773259.946382:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5f80 x1862032664264576/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773300 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773259.946444:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946458:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663677444/109 (0 0) 00010000:00010000:2.0:1775773259.946467:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946475:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.946577:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200000402:0x72b:0x0], updating i_size 8192 00000080:00010000:2.0:1775773259.946591:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000402:0x72b:0x0](ffff98296fae8908) for lock 0x12ae90f9442ebb44 00000080:00010000:2.0:1775773259.946604:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x72b:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773259.946608:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb44 from it ffffa414081d3b20 00010000:00010000:2.0:1775773259.946611:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.946619:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1775773259.946653:0:412860:0:(file.c:6226:ll_getattr_dentry()) START file d60.sanityn:[0x200000402:0x72b:0x0](ffff98296fae8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775773259.946659:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x72b:0x0](ffff98296fae8908),name=d60.sanityn 00000002:00010000:2.0:1775773259.946684:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000402:0x72b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946693:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946703:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1775773259.946712:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd0:0x6d:0x0]) in obj [0x200000bd0:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946718:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946725:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946733:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946741:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1775773259.946751:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000404:0x6d:0x0]) in obj [0x240000404:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946757:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946764:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946772:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946779:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775773259.946788:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1775773259.946792:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x72b:0x0](ffff98296fae8908) 00000080:00010000:2.0:1775773259.946796:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000402:0x72b:0x0](ffff98296fae8908) for lock 0x12ae90f9442ebb44 00000080:00010000:2.0:1775773259.946799:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb44 from it ffffa414081d3cb8 00010000:00010000:2.0:1775773259.946802:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946808:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebb44 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x72b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4659 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1775773259.946819:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd0:0x6d:0x0]) in obj [0x200000bd0:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946824:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946831:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946851:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 3/1,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946857:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ebb4b lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4667 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1775773259.946866:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000404:0x6d:0x0]) in obj [0x240000404:0x6d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946871:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946884:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946890:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775773259.946897:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7800/0x12ae90f9442ebb52 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4675 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1775773259.946948:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988),name=/ 00000002:00010000:2.0:1775773259.946956:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773259.946962:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946969:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773259.946976:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775773259.946980:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) 00000080:00010000:2.0:1775773259.946982:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) for lock 0x12ae90f9442ebb3d 00000080:00010000:2.0:1775773259.946985:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebb3d from it ffffa414081d3bf8 00010000:00010000:2.0:1775773259.946988:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773259.946994:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6600/0x12ae90f9442ebb3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c463d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775773259.947006:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988), inode mode 41ff mask 1 00000080:00200000:2.0:1775773259.947012:0:412860:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60.sanityn, flags=266 00000080:00200000:2.0:1775773259.947017:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000402:0x72b:0x0](ffff98296fae8908), inode mode 41ed mask 44 00000080:00200008:2.0:1775773259.947025:0:412860:0:(file.c:1057:ll_file_open()) START file d60.sanityn:[0x200000402:0x72b:0x0](ffff98296fae8908), flags 704400 00000002:00010000:2.0:1775773259.947039:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000402:0x72b:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1775773259.947110:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ebb59 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773259.947117:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ebb59 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775773259.947127:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775773259.947134:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5180 x1862032664264832/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 864/132072 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775773259.947142:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ebb59 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773259.947163:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664264832:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773259.947604:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d0c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773260.016270:0:404323:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773260.118886:0:404323:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773260.119400:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c35880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664269312:192.168.201.118@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775773260.226305:0:404324:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773260.361320:0:404323:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773260.367535:0:404323:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00000100:00100000:2.0:1775773260.370626:0:404324:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00010000:00010000:2.0:1775773260.488952:0:412926:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebc8d lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x178:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 412926 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.488961:0:412926:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6400/0x12ae90f9442ebc8d lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x178:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 412926 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773260.682956:0:404825:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862017593007232 00000100:00100000:2.0:1775773260.682967:0:404825:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862017593007232 00000100:00100000:2.0:1775773260.683019:0:404825:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.118@tcp, seq: 292 00000100:00100000:2.0:1775773260.683029:0:404825:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff982981c35880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:14dd209b-d605-4024-a2b7-27ec435c0914+5:223164:x1862017593007232:12345-192.168.201.118@tcp:104: 00010000:00010000:2.0:1775773260.683047:0:404825:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebcf6 lrc: 2/0,0 mode: PR/PR res: [0x240000bd2:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96e4bd9bff2c4df3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1775773260.683321:0:404825:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1775773260.683328:0:404825:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff982981c35880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:14dd209b-d605-4024-a2b7-27ec435c0914+5:223164:x1862017593007232:12345-192.168.201.118@tcp:104: Request processed in 300us (447us total) trans 0 rc 0/0 00000100:00100000:2.0:1775773260.683336:0:404825:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.118@tcp, seq: 292 00000100:00100000:2.0:1775773260.696242:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bf800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664289920:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773260.696264:0:412860:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696278:0:412860:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696285:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696292:0:412860:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 3/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696306:0:412860:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696312:0:412860:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff98294166b800 already unused, calling callback (ffffffffc1b072e0) 00010000:00010000:2.0:1775773260.696317:0:412860:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696324:0:412860:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 4/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696332:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 5/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696339:0:412860:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775773260.696355:0:412860:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 3/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773260.696362:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b800/0x12ae90f9442ebd27 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773260.696381:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9829705bf800 x1862032664289920/t12884953521(12884953521) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 864/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775773260.696393:0:412860:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9829705bf800 x1862032664289920/t12884953521(12884953521) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 864/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775773260.696409:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x2a8:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1775773260.696441:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x2000013a1:0x2aa:0x0], updating i_size 4096 00000002:00100000:2.0:1775773260.696459:0:412860:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9829705bf800 x1862032664289920/t12884953521(12884953521) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1775773260.696539:0:412860:0:(file.c:6226:ll_getattr_dentry()) START file d1:[0x2000013a1:0x2aa:0x0](ffff98296fae63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775773260.696546:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x2aa:0x0](ffff98296fae63c8),name=d1 00000002:00010000:2.0:1775773260.696564:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a1:0x2aa:0x0]) in obj [0x2000013a1:0x2a8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773260.696579:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696591:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1775773260.696600:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a0:0x1d:0x0]) in obj [0x2000013a0:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773260.696606:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696612:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696621:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 3/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696629:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/0,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1775773260.696642:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd2:0x4:0x0]) in obj [0x240000bd2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773260.696648:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696656:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696664:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 3/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775773260.696671:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/0,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1775773260.696680:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x2a8:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1775773260.696685:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x2aa:0x0](ffff98296fae63c8) 00000080:00010000:2.0:1775773260.696688:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x2aa:0x0](ffff98296fae63c8) for lock 0x12ae90f9442ebcfd 00000080:00010000:2.0:1775773260.696693:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebcfd from it ffffa414081d3cd8 00010000:00010000:2.0:1775773260.696695:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696702:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1775773260.696712:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a0:0x1d:0x0]) in obj [0x2000013a0:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773260.696718:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696726:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696733:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 3/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696740:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/0,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1775773260.696748:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd2:0x4:0x0]) in obj [0x240000bd2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773260.696751:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696757:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696764:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 3/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1775773260.696770:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/0,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1775773260.696794:0:412860:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x2000013a1:0x2aa:0x0](ffff98296fae63c8) pos/size0/8192 32bit_api 0 00010000:00010000:2.0:1775773260.696804:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebce1 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x2a8:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4db4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1775773260.696839:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a0:0x1d:0x0]) in obj [0x2000013a0:0x1d:0x0], intent: readdir flags 0 00010000:00010000:2.0:1775773260.696846:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.696854:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1775773260.697433:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290048:192.168.201.118@tcp:37:rm.0 00000100:00100000:2.0:1775773260.697502:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 11 seconds 00000100:00100000:2.0:1775773260.713590:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290048:192.168.201.118@tcp:37:rm.0 00010000:00010000:2.0:1775773260.714699:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 3/1,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.714716:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/0,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:2.0:1775773260.714740:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd2:0x4:0x0]) in obj [0x240000bd2:0x4:0x0], intent: readdir flags 0 00010000:00010000:2.0:1775773260.714756:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.714769:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1775773260.715435:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290176:192.168.201.118@tcp:37:rm.0 00000100:00100000:2.0:1775773260.715538:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 11 seconds 00000100:00100000:2.0:1775773260.729125:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290176:192.168.201.118@tcp:37:rm.0 00010000:00010000:2.0:1775773260.729770:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 3/1,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.729787:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982981fe5400/0x12ae90f9442ebd12 lrc: 2/0,0 mode: PR/PR res: [0x240000bd2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4e39 expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1775773260.729882:0:412860:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x2000013a1:0x2aa:0x0](ffff98296fae63c8) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1775773260.729901:0:412860:0:(file.c:447:ll_file_release()) START file d1:[0x2000013a1:0x2aa:0x0](ffff98296fae63c8), flags 704000 00000002:00100000:2.0:1775773260.729963:0:412860:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9829705bf800 x1862032664289920/t12884953521(12884953521) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775773260.729981:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775773260.729986:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7800 x1862032664290432/t0(0) o35->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:23/10 lens 392/840 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775773260.730008:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290432:192.168.201.118@tcp:35:rm.0 00000100:00100000:2.0:1775773260.730123:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 11 seconds 00000100:00100000:2.0:1775773260.743709:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290432:192.168.201.118@tcp:35:rm.0 00000002:00100000:2.0:1775773260.743755:0:412860:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9829705bf800 x1862032664289920/t12884953521(12884953521) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775773260.743815:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x2a8:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773260.743840:0:412860:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1, flags=0 00000080:00200000:2.0:1775773260.743846:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x2a8:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773260.743852:0:412860:0:(namei.c:2339:ll_rmdir()) VFS Op:name=d1, dir=[0x2000013a1:0x2a8:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773260.743888:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.743958:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1775773260.743966:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773260.743969:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe4800/0x12ae90f9442ebcfd lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x2aa:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4e1d expref: -99 pid: 412928 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000020:00100000:2.0:1775773260.743997:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773260.744004:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7800 x1862032664290688/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 488/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773260.744041:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664290688:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773260.744375:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d900 going to sleep for 36 seconds 00000100:00100000:2.0:1775773260.749314:0:412873:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1775773260.749333:0:412873:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862017593008384 00000100:00100000:2.0:1775773260.749341:0:412873:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862017593008384 00000100:00100000:2.0:1775773260.749404:0:412873:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.118@tcp, seq: 293 00000100:00100000:2.0:1775773260.749414:0:412873:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff982948c23050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:14dd209b-d605-4024-a2b7-27ec435c0914+5:201190:x1862017593008384:12345-192.168.201.118@tcp:104: 00010000:00010000:2.0:1775773260.749432:0:412873:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982981fe5e00/0x12ae90f9442ebd0b lrc: 2/0,0 mode: PR/PR res: [0x2000013a0:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x96e4bd9bff2c4e2b expref: -99 pid: 400207 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1775773260.749561:0:412873:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1775773260.749568:0:412873:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff982948c23050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:14dd209b-d605-4024-a2b7-27ec435c0914+5:201190:x1862017593008384:12345-192.168.201.118@tcp:104: Request processed in 156us (387us total) trans 0 rc 0/0 00000100:00100000:2.0:1775773260.749577:0:412873:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.118@tcp, seq: 293 00000100:00100000:2.0:1775773260.766912:0:404323:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.118@tcp (source: 12345-192.168.201.118@tcp) 00010000:00010000:1.0:1775773260.880928:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:1.0:1775773260.880950:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.880956:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1775773260.880963:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00200000:1.0:1775773260.880971:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f9de82af4175d623 [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0x6217d50be8a29dc 00010000:00010000:1.0:1775773260.880975:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1775773260.880982:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00010000:1.0:1775773260.881022:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881030:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1775773260.881038:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000002:00200000:1.0:1775773260.881047:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f892c91a2f7a020b [0x0 0x7d8382c5cf30e04], hash 0x76d36e5d085fdf4 00010000:00010000:1.0:1775773260.881050:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1775773260.881057:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00010000:1.0:1775773260.881090:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881096:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1775773260.881103:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00200000:1.0:1775773260.881112:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f892c91a2f7a020b [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0x76d36e5d085fdf4 00010000:00010000:1.0:1775773260.881116:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1775773260.881122:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00010000:1.0:1775773260.881132:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1775773260.881144:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000002:00200000:1.0:1775773260.881152:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f827c7d3a30cf1fb [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0x7d8382c5cf30e04 00010000:00010000:1.0:1775773260.881155:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1775773260.881162:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00010000:1.0:1775773260.881202:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881209:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1775773260.881217:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00200000:1.0:1775773260.881225:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f4e7e1e1357849cb [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0xb181e1eca87b634 00010000:00010000:1.0:1775773260.881228:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1775773260.881234:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00010000:1.0:1775773260.881243:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881248:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1775773260.881254:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000002:00200000:1.0:1775773260.881262:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f4e7e1e1357849cb [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0xb181e1eca87b634 00010000:00010000:1.0:1775773260.881265:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1775773260.881273:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00010000:1.0:1775773260.881290:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881296:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1775773260.881302:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00200000:1.0:1775773260.881312:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f1c118d35bc321e1 [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0xe3ee72ca43cde1e 00010000:00010000:1.0:1775773260.881316:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1775773260.881322:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000002:00010000:1.0:1775773260.881355:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881361:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1775773260.881368:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000002:00200000:1.0:1775773260.881376:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f0dce26e6e749141 [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0xf231d91918b6ebe 00010000:00010000:1.0:1775773260.881380:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1775773260.881386:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000002:00010000:1.0:1775773260.881396:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881401:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1775773260.881407:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000002:00200000:1.0:1775773260.881414:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f0dce26e6e749141 [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0xf231d91918b6ebe 00010000:00010000:1.0:1775773260.881416:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1775773260.881422:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000002:00010000:1.0:1775773260.881430:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881434:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1775773260.881440:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000002:00200000:1.0:1775773260.881447:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f08a7f92f2ba161b [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0xf75806d0d45e9e4 00010000:00010000:1.0:1775773260.881450:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1775773260.881457:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000002:00010000:1.0:1775773260.881878:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881887:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1775773260.881896:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000002:00200000:1.0:1775773260.881905:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ed7030445a97c18d [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0x128fcfbba5683e72 00010000:00010000:1.0:1775773260.881908:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1775773260.881915:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000002:00010000:1.0:1775773260.881925:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881931:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1775773260.881938:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000002:00200000:1.0:1775773260.881946:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ed7030445a97c18d [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0x128fcfbba5683e72 00010000:00010000:1.0:1775773260.881950:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1775773260.881956:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000002:00010000:1.0:1775773260.881974:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.881979:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1775773260.881986:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000002:00200000:1.0:1775773260.881995:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e97bd9fc59cb237e [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x16842603a634dc81 00010000:00010000:1.0:1775773260.881998:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1775773260.882004:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000002:00010000:1.0:1775773260.882039:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882046:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1775773260.882053:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000002:00200000:1.0:1775773260.882062:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e96721153e6460fc [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0x1698deeac19b9f03 00010000:00010000:1.0:1775773260.882106:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1775773260.882114:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000002:00010000:1.0:1775773260.882124:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882129:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775773260.882137:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000002:00200000:1.0:1775773260.882146:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e96721153e6460fc [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x1698deeac19b9f03 00010000:00010000:1.0:1775773260.882148:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775773260.882154:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000002:00010000:1.0:1775773260.882161:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882166:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775773260.882172:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000002:00200000:1.0:1775773260.882180:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e954ef841320f346 [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x16ab107becdf0cb9 00010000:00010000:1.0:1775773260.882182:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775773260.882188:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000002:00010000:1.0:1775773260.882230:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882236:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775773260.882244:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000002:00200000:1.0:1775773260.882253:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e586387a6cceafdd [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x1a79c78593315022 00010000:00010000:1.0:1775773260.882255:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775773260.882261:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000002:00010000:1.0:1775773260.882270:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882274:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775773260.882282:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000002:00200000:1.0:1775773260.882290:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e586387a6cceafdd [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x1a79c78593315022 00010000:00010000:1.0:1775773260.882294:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775773260.882301:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000002:00010000:1.0:1775773260.882318:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882323:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775773260.882329:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000002:00200000:1.0:1775773260.882338:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2539a4535c00c2d [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x1dac65baca3ff3d2 00010000:00010000:1.0:1775773260.882340:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775773260.882347:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000002:00010000:1.0:1775773260.882381:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882388:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775773260.882395:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000002:00200000:1.0:1775773260.882402:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2184b9d9a7e6741 [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x1de7b462658198be 00010000:00010000:1.0:1775773260.882405:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775773260.882412:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000002:00010000:1.0:1775773260.882422:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882427:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1775773260.882434:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000002:00200000:1.0:1775773260.882442:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2184b9d9a7e6741 [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x1de7b462658198be 00010000:00010000:1.0:1775773260.882445:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1775773260.882451:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000002:00010000:1.0:1775773260.882461:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882465:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1775773260.882471:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000002:00200000:1.0:1775773260.882657:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e1d3d20a06b7787b [0x1e2c2df5f9488784 0x24387659c682dccb], hash 0x1e2c2df5f9488784 00010000:00010000:1.0:1775773260.882662:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1775773260.882670:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000002:00010000:1.0:1775773260.882772:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882779:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1775773260.882786:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000002:00200000:1.0:1775773260.882795:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset df064400c1b6597d [0x1e2c2df5f9488784 0x24387659c682dccb], hash 0x20f9bbff3e49a682 00010000:00010000:1.0:1775773260.882798:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1775773260.882804:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000002:00010000:1.0:1775773260.882812:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882825:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1775773260.882832:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000002:00200000:1.0:1775773260.882840:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset df064400c1b6597d [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x20f9bbff3e49a682 00010000:00010000:1.0:1775773260.882843:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1775773260.882849:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000002:00010000:1.0:1775773260.882866:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882871:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1775773260.882879:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000002:00200000:1.0:1775773260.882888:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset dbc789a6397d2334 [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x24387659c682dccb 00010000:00010000:1.0:1775773260.882892:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1775773260.882899:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000002:00010000:1.0:1775773260.882909:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882914:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1775773260.882921:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000002:00200000:1.0:1775773260.882930:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.882933:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1775773260.882940:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000002:00010000:1.0:1775773260.882975:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.882982:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1775773260.882989:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000002:00200000:1.0:1775773260.882998:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.883001:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1775773260.883006:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000002:00010000:1.0:1775773260.883014:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883018:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1775773260.883026:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000002:00200000:1.0:1775773260.883032:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.883035:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1775773260.883041:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000002:00010000:1.0:1775773260.883105:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883111:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1775773260.883119:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000002:00200000:1.0:1775773260.883126:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d7b0d4123baa955f [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x284f2bedc4556aa0 00010000:00010000:1.0:1775773260.883129:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1775773260.883134:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000002:00010000:1.0:1775773260.883142:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883146:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1775773260.883153:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000002:00200000:1.0:1775773260.883161:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d7b0d4123baa955f [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x284f2bedc4556aa0 00010000:00010000:1.0:1775773260.883164:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1775773260.883170:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000002:00010000:1.0:1775773260.883188:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883192:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1775773260.883200:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000002:00200000:1.0:1775773260.883209:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d42ff02894666189 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2bd00fd76b999e76 00010000:00010000:1.0:1775773260.883212:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1775773260.883220:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000002:00010000:1.0:1775773260.883253:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883261:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1775773260.883269:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000002:00200000:1.0:1775773260.883278:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d406f2547a238a2d [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2bf90dab85dc75d2 00010000:00010000:1.0:1775773260.883281:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1775773260.883288:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000002:00010000:1.0:1775773260.883297:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883302:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1775773260.883309:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000002:00200000:1.0:1775773260.883316:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d406f2547a238a2d [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x2bf90dab85dc75d2 00010000:00010000:1.0:1775773260.883319:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1775773260.883325:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000002:00010000:1.0:1775773260.883335:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883339:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1775773260.883345:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000002:00200000:1.0:1775773260.883354:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d3fb03cc13709040 [0x2c04fc33ec8f6fbf 0x330907229a20881b], hash 0x2c04fc33ec8f6fbf 00010000:00010000:1.0:1775773260.883357:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1775773260.883364:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000002:00010000:1.0:1775773260.883404:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883410:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1775773260.883418:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000002:00200000:1.0:1775773260.883427:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d04427cfb77bca80 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2fbbd8304884357f 00010000:00010000:1.0:1775773260.883430:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1775773260.883437:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000002:00010000:1.0:1775773260.883445:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883449:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1775773260.883455:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000002:00200000:1.0:1775773260.883463:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d04427cfb77bca80 [0x2c04fc33ec8f6fbf 0x330907229a20881b], hash 0x2fbbd8304884357f 00010000:00010000:1.0:1775773260.883466:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1775773260.883473:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000002:00010000:1.0:1775773260.883489:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883842:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1775773260.883852:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000002:00200000:1.0:1775773260.883863:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ccf6f8dd65df77e4 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x330907229a20881b 00010000:00010000:1.0:1775773260.883867:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1775773260.883873:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000002:00010000:1.0:1775773260.883907:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883914:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1775773260.883922:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000002:00200000:1.0:1775773260.883931:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cc527fb1163795a6 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x33ad804ee9c86a59 00010000:00010000:1.0:1775773260.883934:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1775773260.883942:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000002:00010000:1.0:1775773260.883951:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883955:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1775773260.883963:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000002:00200000:1.0:1775773260.883971:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cc527fb1163795a6 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x33ad804ee9c86a59 00010000:00010000:1.0:1775773260.883974:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1775773260.883980:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000002:00010000:1.0:1775773260.883991:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.883995:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1775773260.884002:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000002:00200000:1.0:1775773260.884010:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cbe18c06371d43c7 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x341e73f9c8e2bc38 00010000:00010000:1.0:1775773260.884012:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1775773260.884019:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000002:00010000:1.0:1775773260.884153:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884161:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1775773260.884169:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000002:00200000:1.0:1775773260.884178:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c884d80902a57ef3 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x377b27f6fd5a810c 00010000:00010000:1.0:1775773260.884181:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1775773260.884188:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000002:00010000:1.0:1775773260.884196:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884200:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1775773260.884206:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000002:00200000:1.0:1775773260.884214:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c884d80902a57ef3 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x377b27f6fd5a810c 00010000:00010000:1.0:1775773260.884218:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1775773260.884224:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000002:00010000:1.0:1775773260.884240:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884244:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1775773260.884251:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000002:00200000:1.0:1775773260.884261:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c5918d1fe1614662 [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3a6e72e01e9eb99d 00010000:00010000:1.0:1775773260.884264:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1775773260.884271:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000002:00010000:1.0:1775773260.884311:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884317:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1775773260.884326:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000002:00200000:1.0:1775773260.884333:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c458fe4a6f9f23d4 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x3ba701b59060dc2b 00010000:00010000:1.0:1775773260.884336:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1775773260.884344:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000002:00010000:1.0:1775773260.884354:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884358:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1775773260.884366:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000002:00200000:1.0:1775773260.884374:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c458fe4a6f9f23d4 [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3ba701b59060dc2b 00010000:00010000:1.0:1775773260.884378:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1775773260.884384:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000002:00010000:1.0:1775773260.884396:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884401:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1775773260.884407:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000002:00200000:1.0:1775773260.884417:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c2d261ee9e9c06c2 [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x3d2d9e116163f93d 00010000:00010000:1.0:1775773260.884419:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1775773260.884425:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000002:00010000:1.0:1775773260.884460:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884466:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1775773260.884473:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000002:00200000:1.0:1775773260.884482:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c110a8ef13f002ca [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x3eef5710ec0ffd35 00010000:00010000:1.0:1775773260.884484:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1775773260.884490:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000002:00010000:1.0:1775773260.884641:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884647:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1775773260.884655:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000002:00200000:1.0:1775773260.884663:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c110a8ef13f002ca [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3eef5710ec0ffd35 00010000:00010000:1.0:1775773260.884666:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1775773260.884672:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000002:00010000:1.0:1775773260.884687:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884691:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1775773260.884698:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000002:00200000:1.0:1775773260.884707:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bede630a5be94a70 [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x41219cf5a416b58f 00010000:00010000:1.0:1775773260.884802:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1775773260.884810:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000002:00010000:1.0:1775773260.884857:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884864:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1775773260.884873:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000002:00200000:1.0:1775773260.884881:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bd39cd11ba71f9df [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x42c632ee458e0620 00010000:00010000:1.0:1775773260.884885:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1775773260.884892:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000002:00010000:1.0:1775773260.884902:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884906:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1775773260.884913:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000002:00200000:1.0:1775773260.884923:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bd39cd11ba71f9df [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x42c632ee458e0620 00010000:00010000:1.0:1775773260.884926:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1775773260.884933:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000002:00010000:1.0:1775773260.884944:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.884948:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1775773260.884954:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000002:00200000:1.0:1775773260.884963:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bc3a14e2c9fee9f5 [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x43c5eb1d3601160a 00010000:00010000:1.0:1775773260.884965:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1775773260.884971:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000002:00010000:1.0:1775773260.885007:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885014:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1775773260.885022:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000002:00200000:1.0:1775773260.885030:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ba6c2ba14149a96e [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x4593d45ebeb65691 00010000:00010000:1.0:1775773260.885033:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1775773260.885040:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000002:00010000:1.0:1775773260.885048:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885053:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1775773260.885059:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000002:00200000:1.0:1775773260.885155:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ba6c2ba14149a96e [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x4593d45ebeb65691 00010000:00010000:1.0:1775773260.885159:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1775773260.885165:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000002:00010000:1.0:1775773260.885185:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1775773260.885197:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000002:00200000:1.0:1775773260.885207:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b7547e1fbb1bb971 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x48ab81e044e4468e 00010000:00010000:1.0:1775773260.885211:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1775773260.885218:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000002:00010000:1.0:1775773260.885250:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885257:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1775773260.885265:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000002:00200000:1.0:1775773260.885274:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6e9b404e99f9f54 [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x49164bfb166060ab 00010000:00010000:1.0:1775773260.885277:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1775773260.885283:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000002:00010000:1.0:1775773260.885292:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885298:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1775773260.885305:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000002:00200000:1.0:1775773260.885314:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6e9b404e99f9f54 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x49164bfb166060ab 00010000:00010000:1.0:1775773260.885317:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1775773260.885323:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000002:00010000:1.0:1775773260.885332:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885336:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1775773260.885343:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000002:00200000:1.0:1775773260.885352:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6b95dd8701eb371 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x4946a2278fe14c8e 00010000:00010000:1.0:1775773260.885355:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1775773260.885361:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000002:00010000:1.0:1775773260.885401:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885408:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1775773260.885415:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000002:00200000:1.0:1775773260.885424:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b2d0feb70e0805c0 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x4d2f0148f1f7fa3f 00010000:00010000:1.0:1775773260.885427:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1775773260.885433:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000002:00010000:1.0:1775773260.885441:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885446:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1775773260.885453:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000002:00200000:1.0:1775773260.885462:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b2d0feb70e0805c0 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x4d2f0148f1f7fa3f 00010000:00010000:1.0:1775773260.885467:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1775773260.885473:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000002:00010000:1.0:1775773260.885490:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885580:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1775773260.885588:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000002:00200000:1.0:1775773260.885600:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset afdc90cb569b9579 [0x50236f34a9646a86 0x571beecb771c6161], hash 0x50236f34a9646a86 00010000:00010000:1.0:1775773260.885604:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1775773260.885610:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000002:00010000:1.0:1775773260.885645:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885653:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1775773260.885661:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000002:00200000:1.0:1775773260.885669:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aef49428fb096639 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x510b6bd704f699c6 00010000:00010000:1.0:1775773260.885672:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1775773260.885679:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000002:00010000:1.0:1775773260.885688:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885693:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1775773260.885699:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000002:00200000:1.0:1775773260.885707:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aef49428fb096639 [0x50236f34a9646a86 0x571beecb771c6161], hash 0x510b6bd704f699c6 00010000:00010000:1.0:1775773260.885812:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1775773260.885829:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000002:00010000:1.0:1775773260.885842:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885847:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1775773260.885856:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000002:00200000:1.0:1775773260.885866:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset adba5a29b9419939 [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x5245a5d646be66c6 00010000:00010000:1.0:1775773260.885870:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1775773260.885876:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000002:00010000:1.0:1775773260.885915:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885922:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1775773260.885929:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000002:00200000:1.0:1775773260.885940:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aa78d8b1c42a508b [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x5587274e3bd5af74 00010000:00010000:1.0:1775773260.885943:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1775773260.885950:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000002:00010000:1.0:1775773260.885957:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.885962:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1775773260.885969:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000002:00200000:1.0:1775773260.885975:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aa78d8b1c42a508b [0x50236f34a9646a86 0x571beecb771c6161], hash 0x5587274e3bd5af74 00010000:00010000:1.0:1775773260.885978:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1775773260.885984:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000002:00010000:1.0:1775773260.885997:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886000:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1775773260.886006:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000002:00200000:1.0:1775773260.886015:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a8e4113488e39e9e [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x571beecb771c6161 00010000:00010000:1.0:1775773260.886017:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1775773260.886023:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000002:00010000:1.0:1775773260.886056:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886061:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1775773260.886092:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000002:00200000:1.0:1775773260.886101:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a70b07f5ded57042 [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x58f4f80a212a8fbd 00010000:00010000:1.0:1775773260.886104:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1775773260.886110:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000002:00010000:1.0:1775773260.886119:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886124:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1775773260.886131:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000002:00200000:1.0:1775773260.886140:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a70b07f5ded57042 [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x58f4f80a212a8fbd 00010000:00010000:1.0:1775773260.886143:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1775773260.886149:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000002:00010000:1.0:1775773260.886162:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886165:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1775773260.886171:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000002:00200000:1.0:1775773260.886180:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a557a5d922f0a405 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x5aa85a26dd0f5bfa 00010000:00010000:1.0:1775773260.886183:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1775773260.886190:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000002:00010000:1.0:1775773260.886223:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886229:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1775773260.886236:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000002:00200000:1.0:1775773260.886245:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a31f3c9d9af66195 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x5ce0c36265099e6a 00010000:00010000:1.0:1775773260.886248:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1775773260.886254:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000002:00010000:1.0:1775773260.886261:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886265:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1775773260.886271:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000002:00200000:1.0:1775773260.886278:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a31f3c9d9af66195 [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x5ce0c36265099e6a 00010000:00010000:1.0:1775773260.886280:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1775773260.886286:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000002:00010000:1.0:1775773260.886298:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886302:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1775773260.886308:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000002:00200000:1.0:1775773260.886317:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a15fbaa697519273 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x5ea0455968ae6d8c 00010000:00010000:1.0:1775773260.886320:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1775773260.886325:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000002:00010000:1.0:1775773260.886394:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1775773260.886408:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000002:00200000:1.0:1775773260.886416:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9f8809feedcde0a5 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x6077f60112321f5a 00010000:00010000:1.0:1775773260.886419:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1775773260.886425:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000002:00010000:1.0:1775773260.886434:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886438:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1775773260.886444:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000002:00200000:1.0:1775773260.886452:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9f8809feedcde0a5 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x6077f60112321f5a 00010000:00010000:1.0:1775773260.886455:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1775773260.886462:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000002:00010000:1.0:1775773260.886473:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886477:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1775773260.886484:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000002:00200000:1.0:1775773260.886573:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9e610509bdd53dc3 [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x619efaf6422ac23c 00010000:00010000:1.0:1775773260.886578:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1775773260.886586:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000002:00010000:1.0:1775773260.886629:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886637:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1775773260.886645:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000002:00200000:1.0:1775773260.886656:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9b87e9e9510e8373 [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x64781616aef17c8c 00010000:00010000:1.0:1775773260.886660:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1775773260.886666:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000002:00010000:1.0:1775773260.886675:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886680:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1775773260.886687:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000002:00200000:1.0:1775773260.886696:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9b87e9e9510e8373 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x64781616aef17c8c 00010000:00010000:1.0:1775773260.886699:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1775773260.886706:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000002:00010000:1.0:1775773260.886836:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886843:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1775773260.886850:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000002:00200000:1.0:1775773260.886860:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 98c43aeb5485d045 [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x673bc514ab7a2fba 00010000:00010000:1.0:1775773260.886864:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1775773260.886870:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000002:00010000:1.0:1775773260.886921:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886928:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1775773260.886936:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000002:00200000:1.0:1775773260.886945:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 986aeb8cecb41c5a [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x67951473134be3a5 00010000:00010000:1.0:1775773260.886948:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1775773260.886954:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000002:00010000:1.0:1775773260.886963:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.886967:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1775773260.886974:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000002:00200000:1.0:1775773260.886982:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 98528b25faf07bc7 [0x67ad74da050f8438 0x6f4122b736954945], hash 0x67ad74da050f8438 00010000:00010000:1.0:1775773260.886985:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1775773260.886990:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000002:00010000:1.0:1775773260.886998:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887002:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1775773260.887009:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000002:00200000:1.0:1775773260.887016:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 986aeb8cecb41c5a [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x67951473134be3a5 00010000:00010000:1.0:1775773260.887019:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1775773260.887025:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000002:00010000:1.0:1775773260.887169:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887178:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1775773260.887187:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000002:00200000:1.0:1775773260.887198:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 94a69146b4b9bbeb [0x67ad74da050f8438 0x6f4122b736954945], hash 0x6b596eb94b464414 00010000:00010000:1.0:1775773260.887201:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1775773260.887208:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000002:00010000:1.0:1775773260.887218:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887223:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1775773260.887230:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000002:00200000:1.0:1775773260.887237:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 94a69146b4b9bbeb [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x6b596eb94b464414 00010000:00010000:1.0:1775773260.887241:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1775773260.887247:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000002:00010000:1.0:1775773260.887266:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887271:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1775773260.887279:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000002:00200000:1.0:1775773260.887288:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 90bedd48c96ab6ba [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x6f4122b736954945 00010000:00010000:1.0:1775773260.887291:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1775773260.887298:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000002:00010000:1.0:1775773260.887305:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887310:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1775773260.887317:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000002:00200000:1.0:1775773260.887326:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909885e2dfbab1d6 [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x6f677a1d20454e29 00010000:00010000:1.0:1775773260.887330:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1775773260.887336:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000002:00010000:1.0:1775773260.887372:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887379:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1775773260.887387:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000002:00200000:1.0:1775773260.887397:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909900a43d8c6ca3 [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x6f66ff5bc273935c 00010000:00010000:1.0:1775773260.887400:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1775773260.887407:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000002:00010000:1.0:1775773260.887416:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887421:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1775773260.887428:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000002:00200000:1.0:1775773260.887437:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909900a43d8c6ca3 [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x6f66ff5bc273935c 00010000:00010000:1.0:1775773260.887440:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1775773260.887447:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000002:00010000:1.0:1775773260.887459:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887464:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1775773260.887471:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000002:00200000:1.0:1775773260.887477:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909885e2dfbab1d6 [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x6f677a1d20454e29 00010000:00010000:1.0:1775773260.887480:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1775773260.887486:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000002:00010000:1.0:1775773260.887605:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887612:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1775773260.887620:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000002:00200000:1.0:1775773260.887629:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8d44199822001a7a [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x72bbe667ddffe585 00010000:00010000:1.0:1775773260.887633:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1775773260.887640:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000002:00010000:1.0:1775773260.887650:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887655:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:1.0:1775773260.887661:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000002:00200000:1.0:1775773260.887670:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8d44199822001a7a [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x72bbe667ddffe585 00010000:00010000:1.0:1775773260.887672:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:1.0:1775773260.887678:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000002:00010000:1.0:1775773260.887695:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887699:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:1.0:1775773260.887706:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000002:00200000:1.0:1775773260.887813:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a2aa7c499e9176b [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x75d5583b6616e894 00010000:00010000:1.0:1775773260.887826:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:1.0:1775773260.887835:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000002:00010000:1.0:1775773260.887845:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887851:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:1.0:1775773260.887859:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000002:00200000:1.0:1775773260.887869:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.887872:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:1.0:1775773260.887878:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000002:00010000:1.0:1775773260.887935:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887942:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:1.0:1775773260.887951:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000002:00200000:1.0:1775773260.887959:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.887962:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:1.0:1775773260.887968:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000002:00010000:1.0:1775773260.887976:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.887981:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:1.0:1775773260.887989:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000002:00200000:1.0:1775773260.887999:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.888002:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:1.0:1775773260.888009:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000002:00010000:1.0:1775773260.888052:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888059:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:1.0:1775773260.888149:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000002:00200000:1.0:1775773260.888159:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8626c1bb2296b5f2 [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x79d93e44dd694a0d 00010000:00010000:1.0:1775773260.888162:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:1.0:1775773260.888169:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000002:00010000:1.0:1775773260.888178:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888183:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1775773260.888189:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000002:00200000:1.0:1775773260.888198:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8626c1bb2296b5f2 [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x79d93e44dd694a0d 00010000:00010000:1.0:1775773260.888202:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1775773260.888208:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000002:00010000:1.0:1775773260.888226:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888231:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:1.0:1775773260.888237:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000002:00200000:1.0:1775773260.888246:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 835694646d17b652 [0x7ca96b9b92e849ad 0xfffffffffffffffe], hash 0x7ca96b9b92e849ad 00010000:00010000:1.0:1775773260.888249:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:1.0:1775773260.888256:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000002:00010000:1.0:1775773260.888291:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888297:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1775773260.888306:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000002:00200000:1.0:1775773260.888314:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 82e910120907ddb9 [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x7d16efedf6f82246 00010000:00010000:1.0:1775773260.888317:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1775773260.888324:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000002:00010000:1.0:1775773260.888333:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888338:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:1.0:1775773260.888344:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000002:00200000:1.0:1775773260.888351:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 82e910120907ddb9 [0x7ca96b9b92e849ad 0xfffffffffffffffe], hash 0x7d16efedf6f82246 00010000:00010000:1.0:1775773260.888354:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:1.0:1775773260.888359:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000002:00010000:1.0:1775773260.888368:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.888373:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:1.0:1775773260.888380:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000002:00200000:1.0:1775773260.888389:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 827bbef03b0c1d2c [0x7d84410fc4f3e2d3 0xfffffffffffffffe], hash 0x7d84410fc4f3e2d3 00010000:00010000:1.0:1775773260.888392:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:1.0:1775773260.888398:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200008:1.0:1775773260.888460:0:412860:0:(file.c:447:ll_file_release()) START file d81b.sanityn:[0x240000bd0:0x152:0x0](ffff9829525921c8), flags 704000 00000002:00100000:1.0:1775773260.888596:0:412860:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff982977d4c700 x1862032664292480/t8589971152(8589971152) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775773260.888608:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775773260.888612:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4d880 x1862032664292992/t0(0) o35->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775773260.888634:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664292992:192.168.201.118@tcp:35:rm.0 00000100:00100000:1.0:1775773260.888857:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a540 going to sleep for 11 seconds 00000100:00100000:1.0:1775773260.893387:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664292992:192.168.201.118@tcp:35:rm.0 00000002:00100000:1.0:1775773260.893423:0:412860:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff982977d4c700 x1862032664292480/t8589971152(8589971152) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775773260.893518:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988),name=/ 00000002:00010000:1.0:1775773260.893538:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.893557:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ebd35 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f27 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773260.893570:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ebd35 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f27 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775773260.893577:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775773260.893582:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) 00000080:00010000:1.0:1775773260.893585:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff982949999988) for lock 0x12ae90f9442ebd35 00000080:00010000:1.0:1775773260.893592:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebd35 from it ffffa414081d3bf8 00010000:00010000:1.0:1775773260.893595:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ebd35 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f27 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773260.893603:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ebd35 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f27 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1775773260.893616:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff982949999988), inode mode 41ff mask 1 00000080:00200000:1.0:1775773260.893622:0:412860:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d81b.sanityn, flags=266 00000080:00200000:1.0:1775773260.893630:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 44 00000080:00200008:1.0:1775773260.893638:0:412860:0:(file.c:1057:ll_file_open()) START file d81b.sanityn:[0x240000bd0:0x152:0x0](ffff9829525921c8), flags 704400 00000002:00010000:1.0:1775773260.893654:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd0:0x152:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1775773260.893703:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.893722:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775773260.893734:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775773260.893740:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4d880 x1862032664293248/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775773260.893749:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773260.893768:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293248:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773260.893951:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a540 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.897521:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293248:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773260.897567:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.897577:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.897588:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 4/1,0 mode: CR/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.897599:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 4/1,0 mode: CR/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773260.897614:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff982977d4d880 x1862032664293248/t8589971154(8589971154) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 736/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775773260.897625:0:412860:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff982977d4d880 x1862032664293248/t8589971154(8589971154) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 736/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773260.897645:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 3/1,0 mode: CR/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.897654:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 16 res 9663679440/338 (0 0) 00000080:00200000:1.0:1775773260.897660:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775773260.897690:0:412860:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240000bd0:0x152:0x0], updating i_size 4096 00000080:00010000:1.0:1775773260.897695:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x152:0x0](ffff9829525921c8) for lock 0x12ae90f9442ebd5f 00000080:00010000:1.0:1775773260.897702:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebd5f from it ffffa414081d3bd0 00010000:00010000:1.0:1775773260.897706:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 3/1,0 mode: CR/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.897724:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6200/0x12ae90f9442ebd5f lrc: 2/0,0 mode: CR/CR res: [0x240000bd0:0x152:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f89 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1775773260.897746:0:412860:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff982977d4d880 x1862032664293248/t8589971154(8589971154) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775773260.897825:0:412860:0:(file.c:6226:ll_getattr_dentry()) START file d81b.sanityn:[0x240000bd0:0x152:0x0](ffff9829525921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775773260.897832:0:412860:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8),name=d81b.sanityn 00000002:00010000:1.0:1775773260.897852:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd0:0x152:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.897862:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebd43 lrc: 2/1,0 mode: PR/PR res: [0x240000bd0:0x152:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f51 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773260.897873:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebd43 lrc: 2/1,0 mode: PR/PR res: [0x240000bd0:0x152:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f51 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1775773260.897882:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.897889:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:1.0:1775773260.897896:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:1.0:1775773260.897904:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:1.0:1775773260.897911:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000002:00010000:1.0:1775773260.897920:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.897929:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.897936:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.897943:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.897950:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000080:00200000:1.0:1775773260.897959:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1775773260.897965:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x152:0x0](ffff9829525921c8) 00000080:00010000:1.0:1775773260.897969:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x152:0x0](ffff9829525921c8) for lock 0x12ae90f9442ebd43 00000080:00010000:1.0:1775773260.897972:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebd43 from it ffffa414081d3cd8 00010000:00010000:1.0:1775773260.897976:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebd43 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x152:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f51 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773260.897984:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ba00/0x12ae90f9442ebd43 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x152:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f51 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1775773260.897995:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.897999:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.898004:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.898009:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1775773260.898013:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000002:00010000:1.0:1775773260.898017:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.898019:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:1.0:1775773260.898023:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:1.0:1775773260.898027:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:1.0:1775773260.898034:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000080:00200000:1.0:1775773260.898062:0:412860:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8) pos/size0/208896 32bit_api 0 00010000:00010000:1.0:1775773260.898108:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a200/0x12ae90f9442ebd35 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f27 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1775773260.898137:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898144:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:1.0:1775773260.898151:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000002:00200000:1.0:1775773260.898167:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0x7d8382c5cf30e04], hash 0x0 00010000:00010000:1.0:1775773260.898170:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:1.0:1775773260.898177:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000002:00010000:1.0:1775773260.898185:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898188:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1775773260.898194:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000002:00200000:1.0:1775773260.898206:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0x6217d50be8a29dc], hash 0x0 00010000:00010000:1.0:1775773260.898208:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1775773260.898211:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000002:00010000:1.0:1775773260.898260:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898266:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1775773260.898274:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000002:00200000:1.0:1775773260.898286:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset fc2dee42a9dde457 [0x0 0x7d8382c5cf30e04], hash 0x3d211bd56221ba8 00010000:00010000:1.0:1775773260.898289:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1775773260.898296:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000002:00010000:1.0:1775773260.898303:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898306:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1775773260.898311:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000002:00200000:1.0:1775773260.898323:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset fc2dee42a9dde457 [0x0 0x6217d50be8a29dc], hash 0x3d211bd56221ba8 00010000:00010000:1.0:1775773260.898325:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1775773260.898329:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000002:00010000:1.0:1775773260.898340:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898343:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1775773260.898346:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000002:00200000:1.0:1775773260.898352:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f9de82af4175d623 [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0x6217d50be8a29dc 00010000:00010000:1.0:1775773260.898354:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1775773260.898358:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000002:00010000:1.0:1775773260.898378:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898382:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1775773260.898386:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000002:00200000:1.0:1775773260.898392:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f892c91a2f7a020b [0x0 0x7d8382c5cf30e04], hash 0x76d36e5d085fdf4 00010000:00010000:1.0:1775773260.898394:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1775773260.898398:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000002:00010000:1.0:1775773260.898403:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898405:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:1.0:1775773260.898409:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000002:00200000:1.0:1775773260.898414:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f892c91a2f7a020b [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0x76d36e5d085fdf4 00010000:00010000:1.0:1775773260.898416:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:1.0:1775773260.898420:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000002:00010000:1.0:1775773260.898425:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898428:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1775773260.898431:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000002:00200000:1.0:1775773260.898438:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f827c7d3a30cf1fb [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0x7d8382c5cf30e04 00010000:00010000:1.0:1775773260.898440:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1775773260.898444:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000002:00010000:1.0:1775773260.898479:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898486:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:1.0:1775773260.898493:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000002:00200000:1.0:1775773260.898501:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f4e7e1e1357849cb [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0xb181e1eca87b634 00010000:00010000:1.0:1775773260.898504:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:1.0:1775773260.898511:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000002:00010000:1.0:1775773260.898520:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898525:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:1.0:1775773260.898530:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000002:00200000:1.0:1775773260.898537:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f4e7e1e1357849cb [0x6217d50be8a29dc 0xe3ee72ca43cde1e], hash 0xb181e1eca87b634 00010000:00010000:1.0:1775773260.898538:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:1.0:1775773260.898543:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000002:00010000:1.0:1775773260.898555:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898557:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:1.0:1775773260.898561:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000002:00200000:1.0:1775773260.898569:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f1c118d35bc321e1 [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0xe3ee72ca43cde1e 00010000:00010000:1.0:1775773260.898571:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:1.0:1775773260.898574:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000002:00010000:1.0:1775773260.898595:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898599:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:1.0:1775773260.898603:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000002:00200000:1.0:1775773260.898608:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f0dce26e6e749141 [0x7d8382c5cf30e04 0xf75806d0d45e9e4], hash 0xf231d91918b6ebe 00010000:00010000:1.0:1775773260.898610:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:1.0:1775773260.898614:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000002:00010000:1.0:1775773260.898619:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898621:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:1.0:1775773260.898625:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000002:00200000:1.0:1775773260.898630:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f0dce26e6e749141 [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0xf231d91918b6ebe 00010000:00010000:1.0:1775773260.898632:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:1.0:1775773260.898639:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000002:00010000:1.0:1775773260.898647:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898652:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:1.0:1775773260.898659:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000002:00200000:1.0:1775773260.898668:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset f08a7f92f2ba161b [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0xf75806d0d45e9e4 00010000:00010000:1.0:1775773260.898671:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:1.0:1775773260.898678:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000002:00010000:1.0:1775773260.898714:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898719:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:1.0:1775773260.898724:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000002:00200000:1.0:1775773260.898731:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ed7030445a97c18d [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0x128fcfbba5683e72 00010000:00010000:1.0:1775773260.898733:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:1.0:1775773260.898737:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000002:00010000:1.0:1775773260.898743:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898748:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:1.0:1775773260.898754:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000002:00200000:1.0:1775773260.898763:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ed7030445a97c18d [0xe3ee72ca43cde1e 0x16842603a634dc81], hash 0x128fcfbba5683e72 00010000:00010000:1.0:1775773260.898766:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:1.0:1775773260.898772:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000002:00010000:1.0:1775773260.898791:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898795:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:1.0:1775773260.898800:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000002:00200000:1.0:1775773260.898807:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e97bd9fc59cb237e [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x16842603a634dc81 00010000:00010000:1.0:1775773260.898809:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:1.0:1775773260.898813:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000002:00010000:1.0:1775773260.898847:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898851:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:1.0:1775773260.898857:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000002:00200000:1.0:1775773260.898867:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e96721153e6460fc [0xf75806d0d45e9e4 0x16ab107becdf0cb9], hash 0x1698deeac19b9f03 00010000:00010000:1.0:1775773260.898870:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:1.0:1775773260.898877:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000002:00010000:1.0:1775773260.898886:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898891:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:1.0:1775773260.898898:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000002:00200000:1.0:1775773260.898907:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e96721153e6460fc [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x1698deeac19b9f03 00010000:00010000:1.0:1775773260.898910:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:1.0:1775773260.898917:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000002:00010000:1.0:1775773260.898925:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898930:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:1.0:1775773260.898937:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000002:00200000:1.0:1775773260.898947:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e954ef841320f346 [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x16ab107becdf0cb9 00010000:00010000:1.0:1775773260.898951:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:1.0:1775773260.898958:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000002:00010000:1.0:1775773260.898987:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.898991:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:1.0:1775773260.898997:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000002:00200000:1.0:1775773260.899004:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e586387a6cceafdd [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x1a79c78593315022 00010000:00010000:1.0:1775773260.899006:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:1.0:1775773260.899010:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000002:00010000:1.0:1775773260.899015:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899017:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1775773260.899021:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000002:00200000:1.0:1775773260.899027:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e586387a6cceafdd [0x16842603a634dc81 0x1dac65baca3ff3d2], hash 0x1a79c78593315022 00010000:00010000:1.0:1775773260.899029:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1775773260.899032:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000002:00010000:1.0:1775773260.899044:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899047:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:1.0:1775773260.899050:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000002:00200000:1.0:1775773260.899056:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2539a4535c00c2d [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x1dac65baca3ff3d2 00010000:00010000:1.0:1775773260.899057:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:1.0:1775773260.899061:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000002:00010000:1.0:1775773260.899107:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899112:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1775773260.899117:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000002:00200000:1.0:1775773260.899123:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2184b9d9a7e6741 [0x16ab107becdf0cb9 0x1e2c2df5f9488784], hash 0x1de7b462658198be 00010000:00010000:1.0:1775773260.899125:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1775773260.899129:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000002:00010000:1.0:1775773260.899135:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1775773260.899141:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000002:00200000:1.0:1775773260.899147:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e2184b9d9a7e6741 [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x1de7b462658198be 00010000:00010000:1.0:1775773260.899149:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1775773260.899152:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000002:00010000:1.0:1775773260.899157:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899159:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:1.0:1775773260.899163:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000002:00200000:1.0:1775773260.899171:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset e1d3d20a06b7787b [0x1e2c2df5f9488784 0x24387659c682dccb], hash 0x1e2c2df5f9488784 00010000:00010000:1.0:1775773260.899174:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:1.0:1775773260.899181:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000002:00010000:1.0:1775773260.899216:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899224:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1775773260.899230:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000002:00200000:1.0:1775773260.899240:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset df064400c1b6597d [0x1e2c2df5f9488784 0x24387659c682dccb], hash 0x20f9bbff3e49a682 00010000:00010000:1.0:1775773260.899243:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1775773260.899249:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000002:00010000:1.0:1775773260.899258:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899262:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:1.0:1775773260.899269:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000002:00200000:1.0:1775773260.899277:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset df064400c1b6597d [0x1dac65baca3ff3d2 0x246bd80b206facbb], hash 0x20f9bbff3e49a682 00010000:00010000:1.0:1775773260.899280:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:1.0:1775773260.899286:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000002:00010000:1.0:1775773260.899301:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899306:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:1.0:1775773260.899312:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000002:00200000:1.0:1775773260.899320:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset dbc789a6397d2334 [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x24387659c682dccb 00010000:00010000:1.0:1775773260.899323:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:1.0:1775773260.899328:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000002:00010000:1.0:1775773260.899337:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899341:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:1.0:1775773260.899347:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000002:00200000:1.0:1775773260.899356:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.899358:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:1.0:1775773260.899365:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000002:00010000:1.0:1775773260.899397:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899404:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:1.0:1775773260.899412:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000002:00200000:1.0:1775773260.899422:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.899424:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:1.0:1775773260.899431:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000002:00010000:1.0:1775773260.899439:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899444:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1775773260.899451:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000002:00200000:1.0:1775773260.899459:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset db9427f4df905344 [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x246bd80b206facbb 00010000:00010000:1.0:1775773260.899462:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1775773260.899469:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000002:00010000:1.0:1775773260.899512:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899520:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1775773260.899528:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000002:00200000:1.0:1775773260.899537:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d7b0d4123baa955f [0x24387659c682dccb 0x2bd00fd76b999e76], hash 0x284f2bedc4556aa0 00010000:00010000:1.0:1775773260.899539:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1775773260.899546:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000002:00010000:1.0:1775773260.899554:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899559:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:1.0:1775773260.899565:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000002:00200000:1.0:1775773260.899575:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d7b0d4123baa955f [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x284f2bedc4556aa0 00010000:00010000:1.0:1775773260.899577:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:1.0:1775773260.899583:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000002:00010000:1.0:1775773260.899600:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899605:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:1.0:1775773260.899612:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000002:00200000:1.0:1775773260.899622:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d42ff02894666189 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2bd00fd76b999e76 00010000:00010000:1.0:1775773260.899625:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:1.0:1775773260.899633:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000002:00010000:1.0:1775773260.899666:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899673:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:1.0:1775773260.899680:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000002:00200000:1.0:1775773260.899690:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d406f2547a238a2d [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2bf90dab85dc75d2 00010000:00010000:1.0:1775773260.899693:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:1.0:1775773260.899700:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000002:00010000:1.0:1775773260.899709:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899714:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:1.0:1775773260.899720:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000002:00200000:1.0:1775773260.899727:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d406f2547a238a2d [0x246bd80b206facbb 0x2c04fc33ec8f6fbf], hash 0x2bf90dab85dc75d2 00010000:00010000:1.0:1775773260.899730:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:1.0:1775773260.899736:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000002:00010000:1.0:1775773260.899747:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899752:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1775773260.899760:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000002:00200000:1.0:1775773260.899770:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d3fb03cc13709040 [0x2c04fc33ec8f6fbf 0x330907229a20881b], hash 0x2c04fc33ec8f6fbf 00010000:00010000:1.0:1775773260.899773:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1775773260.899780:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000002:00010000:1.0:1775773260.899839:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899847:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1775773260.899855:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000002:00200000:1.0:1775773260.899864:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d04427cfb77bca80 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x2fbbd8304884357f 00010000:00010000:1.0:1775773260.899867:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1775773260.899874:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000002:00010000:1.0:1775773260.899884:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899889:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1775773260.899895:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000002:00200000:1.0:1775773260.899907:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset d04427cfb77bca80 [0x2c04fc33ec8f6fbf 0x330907229a20881b], hash 0x2fbbd8304884357f 00010000:00010000:1.0:1775773260.899910:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1775773260.899917:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000002:00010000:1.0:1775773260.899934:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.899940:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1775773260.899947:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000002:00200000:1.0:1775773260.899956:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ccf6f8dd65df77e4 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x330907229a20881b 00010000:00010000:1.0:1775773260.899959:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1775773260.899966:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000002:00010000:1.0:1775773260.900000:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900007:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1775773260.900014:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000002:00200000:1.0:1775773260.900022:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cc527fb1163795a6 [0x2bd00fd76b999e76 0x341e73f9c8e2bc38], hash 0x33ad804ee9c86a59 00010000:00010000:1.0:1775773260.900024:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1775773260.900030:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000002:00010000:1.0:1775773260.900037:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900041:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:1.0:1775773260.900047:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000002:00200000:1.0:1775773260.900055:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cc527fb1163795a6 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x33ad804ee9c86a59 00010000:00010000:1.0:1775773260.900058:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:1.0:1775773260.900092:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000002:00010000:1.0:1775773260.900104:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900109:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1775773260.900116:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000002:00200000:1.0:1775773260.900126:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset cbe18c06371d43c7 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x341e73f9c8e2bc38 00010000:00010000:1.0:1775773260.900129:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1775773260.900135:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000002:00010000:1.0:1775773260.900173:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900180:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:1.0:1775773260.900188:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000002:00200000:1.0:1775773260.900196:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c884d80902a57ef3 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x377b27f6fd5a810c 00010000:00010000:1.0:1775773260.900199:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:1.0:1775773260.900205:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000002:00010000:1.0:1775773260.900214:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900220:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1775773260.900227:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000002:00200000:1.0:1775773260.900235:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c884d80902a57ef3 [0x330907229a20881b 0x3a6e72e01e9eb99d], hash 0x377b27f6fd5a810c 00010000:00010000:1.0:1775773260.900237:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1775773260.900243:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000002:00010000:1.0:1775773260.900257:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900262:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1775773260.900269:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000002:00200000:1.0:1775773260.900279:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c5918d1fe1614662 [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3a6e72e01e9eb99d 00010000:00010000:1.0:1775773260.900281:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1775773260.900288:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000002:00010000:1.0:1775773260.900322:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900330:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1775773260.900338:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000002:00200000:1.0:1775773260.900346:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c458fe4a6f9f23d4 [0x341e73f9c8e2bc38 0x3d2d9e116163f93d], hash 0x3ba701b59060dc2b 00010000:00010000:1.0:1775773260.900349:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1775773260.900357:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000002:00010000:1.0:1775773260.900366:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900371:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1775773260.900379:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000002:00200000:1.0:1775773260.900388:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c458fe4a6f9f23d4 [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3ba701b59060dc2b 00010000:00010000:1.0:1775773260.900391:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1775773260.900398:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000002:00010000:1.0:1775773260.900408:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900412:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1775773260.900419:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000002:00200000:1.0:1775773260.900430:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c2d261ee9e9c06c2 [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x3d2d9e116163f93d 00010000:00010000:1.0:1775773260.900434:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1775773260.900442:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000002:00010000:1.0:1775773260.900480:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.900487:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1775773260.900495:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000002:00200000:1.0:1775773260.900505:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c110a8ef13f002ca [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x3eef5710ec0ffd35 00010000:00010000:1.0:1775773260.900508:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1775773260.900514:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000002:00010000:1.0:1775773260.900523:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901237:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1775773260.901250:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000002:00200000:1.0:1775773260.901260:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset c110a8ef13f002ca [0x3a6e72e01e9eb99d 0x41219cf5a416b58f], hash 0x3eef5710ec0ffd35 00010000:00010000:1.0:1775773260.901264:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1775773260.901270:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000002:00010000:1.0:1775773260.901288:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901294:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:1.0:1775773260.901301:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000002:00200000:1.0:1775773260.901310:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bede630a5be94a70 [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x41219cf5a416b58f 00010000:00010000:1.0:1775773260.901312:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:1.0:1775773260.901319:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000002:00010000:1.0:1775773260.901359:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901366:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1775773260.901375:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000002:00200000:1.0:1775773260.901385:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bd39cd11ba71f9df [0x3d2d9e116163f93d 0x43c5eb1d3601160a], hash 0x42c632ee458e0620 00010000:00010000:1.0:1775773260.901389:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1775773260.901396:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000002:00010000:1.0:1775773260.901404:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901409:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1775773260.901416:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000002:00200000:1.0:1775773260.901425:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bd39cd11ba71f9df [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x42c632ee458e0620 00010000:00010000:1.0:1775773260.901429:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1775773260.901435:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000002:00010000:1.0:1775773260.901446:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901450:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:1.0:1775773260.901457:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000002:00200000:1.0:1775773260.901466:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset bc3a14e2c9fee9f5 [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x43c5eb1d3601160a 00010000:00010000:1.0:1775773260.901469:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:1.0:1775773260.901475:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000002:00010000:1.0:1775773260.901509:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901516:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1775773260.901522:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000002:00200000:1.0:1775773260.901587:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ba6c2ba14149a96e [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x4593d45ebeb65691 00010000:00010000:1.0:1775773260.901590:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1775773260.901597:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000002:00010000:1.0:1775773260.901608:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901614:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:1.0:1775773260.901621:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000002:00200000:1.0:1775773260.901629:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset ba6c2ba14149a96e [0x41219cf5a416b58f 0x48ab81e044e4468e], hash 0x4593d45ebeb65691 00010000:00010000:1.0:1775773260.901633:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:1.0:1775773260.901638:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000002:00010000:1.0:1775773260.901657:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901662:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:1.0:1775773260.901669:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000002:00200000:1.0:1775773260.901678:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b7547e1fbb1bb971 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x48ab81e044e4468e 00010000:00010000:1.0:1775773260.901681:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:1.0:1775773260.901688:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000002:00010000:1.0:1775773260.901834:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901842:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:1.0:1775773260.901850:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000002:00200000:1.0:1775773260.901859:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6e9b404e99f9f54 [0x43c5eb1d3601160a 0x4946a2278fe14c8e], hash 0x49164bfb166060ab 00010000:00010000:1.0:1775773260.901863:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:1.0:1775773260.901870:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000002:00010000:1.0:1775773260.901879:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901885:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1775773260.901892:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000002:00200000:1.0:1775773260.901902:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6e9b404e99f9f54 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x49164bfb166060ab 00010000:00010000:1.0:1775773260.901905:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1775773260.901911:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000002:00010000:1.0:1775773260.901919:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901924:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:1.0:1775773260.901930:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000002:00200000:1.0:1775773260.901938:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b6b95dd8701eb371 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x4946a2278fe14c8e 00010000:00010000:1.0:1775773260.901941:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:1.0:1775773260.901947:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000002:00010000:1.0:1775773260.901984:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.901990:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1775773260.901997:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000002:00200000:1.0:1775773260.902006:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b2d0feb70e0805c0 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x4d2f0148f1f7fa3f 00010000:00010000:1.0:1775773260.902008:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1775773260.902014:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000002:00010000:1.0:1775773260.902022:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902026:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:1.0:1775773260.902032:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000002:00200000:1.0:1775773260.902040:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset b2d0feb70e0805c0 [0x48ab81e044e4468e 0x50236f34a9646a86], hash 0x4d2f0148f1f7fa3f 00010000:00010000:1.0:1775773260.902043:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:1.0:1775773260.902049:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000002:00010000:1.0:1775773260.902152:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902159:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1775773260.902166:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000002:00200000:1.0:1775773260.902175:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset afdc90cb569b9579 [0x50236f34a9646a86 0x571beecb771c6161], hash 0x50236f34a9646a86 00010000:00010000:1.0:1775773260.902178:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1775773260.902185:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000002:00010000:1.0:1775773260.902223:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902230:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:1.0:1775773260.902237:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000002:00200000:1.0:1775773260.902246:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aef49428fb096639 [0x4946a2278fe14c8e 0x5245a5d646be66c6], hash 0x510b6bd704f699c6 00010000:00010000:1.0:1775773260.902249:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:1.0:1775773260.902256:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000002:00010000:1.0:1775773260.902264:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902268:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1775773260.902275:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000002:00200000:1.0:1775773260.902282:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aef49428fb096639 [0x50236f34a9646a86 0x571beecb771c6161], hash 0x510b6bd704f699c6 00010000:00010000:1.0:1775773260.902285:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1775773260.902292:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000002:00010000:1.0:1775773260.902302:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902306:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1775773260.902312:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000002:00200000:1.0:1775773260.902321:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset adba5a29b9419939 [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x5245a5d646be66c6 00010000:00010000:1.0:1775773260.902324:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1775773260.902331:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000002:00010000:1.0:1775773260.902367:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902373:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1775773260.902379:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000002:00200000:1.0:1775773260.902389:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aa78d8b1c42a508b [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x5587274e3bd5af74 00010000:00010000:1.0:1775773260.902392:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1775773260.902398:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000002:00010000:1.0:1775773260.902407:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902411:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:1.0:1775773260.902418:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000002:00200000:1.0:1775773260.902426:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset aa78d8b1c42a508b [0x50236f34a9646a86 0x571beecb771c6161], hash 0x5587274e3bd5af74 00010000:00010000:1.0:1775773260.902429:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:1.0:1775773260.902435:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000002:00010000:1.0:1775773260.902449:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902454:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:1.0:1775773260.902461:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000002:00200000:1.0:1775773260.902470:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a8e4113488e39e9e [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x571beecb771c6161 00010000:00010000:1.0:1775773260.902474:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:1.0:1775773260.902480:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000002:00010000:1.0:1775773260.902516:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902523:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:1.0:1775773260.902599:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000002:00200000:1.0:1775773260.902609:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a70b07f5ded57042 [0x5245a5d646be66c6 0x5aa85a26dd0f5bfa], hash 0x58f4f80a212a8fbd 00010000:00010000:1.0:1775773260.902612:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:1.0:1775773260.902619:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000002:00010000:1.0:1775773260.902627:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902633:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:1.0:1775773260.902640:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000002:00200000:1.0:1775773260.902650:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a70b07f5ded57042 [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x58f4f80a212a8fbd 00010000:00010000:1.0:1775773260.902652:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:1.0:1775773260.902659:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000002:00010000:1.0:1775773260.902672:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902677:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:1.0:1775773260.902685:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000002:00200000:1.0:1775773260.902695:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a557a5d922f0a405 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x5aa85a26dd0f5bfa 00010000:00010000:1.0:1775773260.902698:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:1.0:1775773260.902704:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000002:00010000:1.0:1775773260.902846:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902854:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:1.0:1775773260.902862:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000002:00200000:1.0:1775773260.902872:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a31f3c9d9af66195 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x5ce0c36265099e6a 00010000:00010000:1.0:1775773260.902875:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:1.0:1775773260.902881:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000002:00010000:1.0:1775773260.902890:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902894:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:1.0:1775773260.902901:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000002:00200000:1.0:1775773260.902911:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a31f3c9d9af66195 [0x571beecb771c6161 0x5ea0455968ae6d8c], hash 0x5ce0c36265099e6a 00010000:00010000:1.0:1775773260.902913:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:1.0:1775773260.902919:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000002:00010000:1.0:1775773260.902931:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902935:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:1.0:1775773260.902940:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000002:00200000:1.0:1775773260.902948:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset a15fbaa697519273 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x5ea0455968ae6d8c 00010000:00010000:1.0:1775773260.902950:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:1.0:1775773260.902956:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000002:00010000:1.0:1775773260.902989:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.902996:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:1.0:1775773260.903004:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000002:00200000:1.0:1775773260.903013:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9f8809feedcde0a5 [0x5aa85a26dd0f5bfa 0x619efaf6422ac23c], hash 0x6077f60112321f5a 00010000:00010000:1.0:1775773260.903016:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:1.0:1775773260.903023:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000002:00010000:1.0:1775773260.903031:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903037:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1775773260.903044:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000002:00200000:1.0:1775773260.903053:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9f8809feedcde0a5 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x6077f60112321f5a 00010000:00010000:1.0:1775773260.903057:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1775773260.903063:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000002:00010000:1.0:1775773260.903163:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903169:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:1.0:1775773260.903177:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000002:00200000:1.0:1775773260.903189:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9e610509bdd53dc3 [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x619efaf6422ac23c 00010000:00010000:1.0:1775773260.903193:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:1.0:1775773260.903200:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000002:00010000:1.0:1775773260.903238:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903245:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1775773260.903253:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000002:00200000:1.0:1775773260.903262:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9b87e9e9510e8373 [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x64781616aef17c8c 00010000:00010000:1.0:1775773260.903265:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1775773260.903272:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000002:00010000:1.0:1775773260.903280:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903285:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:1.0:1775773260.903293:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000002:00200000:1.0:1775773260.903302:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 9b87e9e9510e8373 [0x5ea0455968ae6d8c 0x673bc514ab7a2fba], hash 0x64781616aef17c8c 00010000:00010000:1.0:1775773260.903305:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:1.0:1775773260.903311:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000002:00010000:1.0:1775773260.903328:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903334:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:1.0:1775773260.903341:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000002:00200000:1.0:1775773260.903350:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 98c43aeb5485d045 [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x673bc514ab7a2fba 00010000:00010000:1.0:1775773260.903353:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:1.0:1775773260.903359:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000002:00010000:1.0:1775773260.903392:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903398:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:1.0:1775773260.903406:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000002:00200000:1.0:1775773260.903415:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 986aeb8cecb41c5a [0x619efaf6422ac23c 0x67ad74da050f8438], hash 0x67951473134be3a5 00010000:00010000:1.0:1775773260.903418:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:1.0:1775773260.903424:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000002:00010000:1.0:1775773260.903432:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903436:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:1.0:1775773260.903444:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000002:00200000:1.0:1775773260.903453:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 98528b25faf07bc7 [0x67ad74da050f8438 0x6f4122b736954945], hash 0x67ad74da050f8438 00010000:00010000:1.0:1775773260.903456:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:1.0:1775773260.903463:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000002:00010000:1.0:1775773260.903472:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903477:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1775773260.903483:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000002:00200000:1.0:1775773260.903492:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 986aeb8cecb41c5a [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x67951473134be3a5 00010000:00010000:1.0:1775773260.903495:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1775773260.903501:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000002:00010000:1.0:1775773260.903619:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903627:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1775773260.903636:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000002:00200000:1.0:1775773260.903647:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 94a69146b4b9bbeb [0x67ad74da050f8438 0x6f4122b736954945], hash 0x6b596eb94b464414 00010000:00010000:1.0:1775773260.903649:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1775773260.903655:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000002:00010000:1.0:1775773260.903663:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903667:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:1.0:1775773260.903673:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000002:00200000:1.0:1775773260.903680:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 94a69146b4b9bbeb [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x6b596eb94b464414 00010000:00010000:1.0:1775773260.903682:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:1.0:1775773260.903688:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000002:00010000:1.0:1775773260.903704:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903708:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:1.0:1775773260.903805:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000002:00200000:1.0:1775773260.903814:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 90bedd48c96ab6ba [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x6f4122b736954945 00010000:00010000:1.0:1775773260.903826:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:1.0:1775773260.903833:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000002:00010000:1.0:1775773260.903842:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903847:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:1.0:1775773260.903855:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000002:00200000:1.0:1775773260.903865:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909885e2dfbab1d6 [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x6f677a1d20454e29 00010000:00010000:1.0:1775773260.903868:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:1.0:1775773260.903875:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000002:00010000:1.0:1775773260.903905:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903911:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:1.0:1775773260.903919:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000002:00200000:1.0:1775773260.903928:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909900a43d8c6ca3 [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x6f66ff5bc273935c 00010000:00010000:1.0:1775773260.903930:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:1.0:1775773260.903936:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000002:00010000:1.0:1775773260.903943:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903947:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:1.0:1775773260.903953:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000002:00200000:1.0:1775773260.903960:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909900a43d8c6ca3 [0x673bc514ab7a2fba 0x6f677a1d20454e29], hash 0x6f66ff5bc273935c 00010000:00010000:1.0:1775773260.903963:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:1.0:1775773260.903969:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00000002:00010000:1.0:1775773260.903977:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.903980:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00010000:00010000:1.0:1775773260.903987:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00000002:00200000:1.0:1775773260.903992:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 909885e2dfbab1d6 [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x6f677a1d20454e29 00010000:00010000:1.0:1775773260.903995:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00010000:00010000:1.0:1775773260.904001:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00000002:00010000:1.0:1775773260.904035:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904042:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:1.0:1775773260.904049:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000002:00200000:1.0:1775773260.904057:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8d44199822001a7a [0x6f4122b736954945 0x75e92b2c3aa2e0a1], hash 0x72bbe667ddffe585 00010000:00010000:1.0:1775773260.904060:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:1.0:1775773260.904150:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00000002:00010000:1.0:1775773260.904159:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904164:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00010000:00010000:1.0:1775773260.904170:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00000002:00200000:1.0:1775773260.904180:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8d44199822001a7a [0x6f677a1d20454e29 0x75d5583b6616e894], hash 0x72bbe667ddffe585 00010000:00010000:1.0:1775773260.904182:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00010000:00010000:1.0:1775773260.904189:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00000002:00010000:1.0:1775773260.904205:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904209:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:1.0:1775773260.904216:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00000002:00200000:1.0:1775773260.904226:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a2aa7c499e9176b [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x75d5583b6616e894 00010000:00010000:1.0:1775773260.904230:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:1.0:1775773260.904237:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000002:00010000:1.0:1775773260.904244:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904249:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00010000:00010000:1.0:1775773260.904256:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00000002:00200000:1.0:1775773260.904265:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.904269:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 106 lru_type: 1 00010000:00010000:1.0:1775773260.904275:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00000002:00010000:1.0:1775773260.904303:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904311:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00010000:00010000:1.0:1775773260.904318:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00000002:00200000:1.0:1775773260.904327:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.904330:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 107 lru_type: 1 00010000:00010000:1.0:1775773260.904337:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00000002:00010000:1.0:1775773260.904345:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904349:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:1.0:1775773260.904356:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000002:00200000:1.0:1775773260.904365:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8a16d4d3c55d1f5e [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x75e92b2c3aa2e0a1 00010000:00010000:1.0:1775773260.904369:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:1.0:1775773260.904375:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00000002:00010000:1.0:1775773260.904414:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904420:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:1.0:1775773260.904427:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00000002:00200000:1.0:1775773260.904437:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8626c1bb2296b5f2 [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x79d93e44dd694a0d 00010000:00010000:1.0:1775773260.904440:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 108 lru_type: 1 00010000:00010000:1.0:1775773260.904447:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000002:00010000:1.0:1775773260.904456:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904460:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00010000:00010000:1.0:1775773260.904467:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00000002:00200000:1.0:1775773260.904475:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 8626c1bb2296b5f2 [0x75d5583b6616e894 0x7ca96b9b92e849ad], hash 0x79d93e44dd694a0d 00010000:00010000:1.0:1775773260.904478:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00010000:00010000:1.0:1775773260.904484:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 111 lru_type: 1 00000002:00010000:1.0:1775773260.904499:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904504:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 111 lru_type: 1 00010000:00010000:1.0:1775773260.904510:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 111 lru_type: 1 00000002:00200000:1.0:1775773260.904520:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 835694646d17b652 [0x7ca96b9b92e849ad 0xfffffffffffffffe], hash 0x7ca96b9b92e849ad 00010000:00010000:1.0:1775773260.904523:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 111 lru_type: 1 00010000:00010000:1.0:1775773260.904815:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 112 lru_type: 1 00000002:00010000:1.0:1775773260.904866:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904874:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:1.0:1775773260.904882:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000002:00200000:1.0:1775773260.904892:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 82e910120907ddb9 [0x75e92b2c3aa2e0a1 0x7d84410fc4f3e2d3], hash 0x7d16efedf6f82246 00010000:00010000:1.0:1775773260.904896:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:1.0:1775773260.904902:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00000002:00010000:1.0:1775773260.904912:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000013a2:0x4:0x0]) in obj [0x2000013a2:0x4:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904917:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 112 lru_type: 1 00010000:00010000:1.0:1775773260.904924:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 112 lru_type: 1 00000002:00200000:1.0:1775773260.904933:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 82e910120907ddb9 [0x7ca96b9b92e849ad 0xfffffffffffffffe], hash 0x7d16efedf6f82246 00010000:00010000:1.0:1775773260.904937:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 3/1,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 112 lru_type: 1 00010000:00010000:1.0:1775773260.904943:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 113 lru_type: 1 00000002:00010000:1.0:1775773260.904951:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240000bd1:0x1e:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775773260.904955:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00010000:00010000:1.0:1775773260.904962:0:412860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00000002:00200000:1.0:1775773260.904970:0:412860:0:(mdc_request.c:1174:mdc_page_locate()) offset 827bbef03b0c1d2c [0x7d84410fc4f3e2d3 0xfffffffffffffffe], hash 0x7d84410fc4f3e2d3 00010000:00010000:1.0:1775773260.904973:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 3/1,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00010000:00010000:1.0:1775773260.904979:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166ac00/0x12ae90f9442ebd4a lrc: 2/0,0 mode: PR/PR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4f5f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 111 lru_type: 1 00000080:00200000:1.0:1775773260.907163:0:412860:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8) pos/size18446744073709551614/208896 32bit_api 0 00000080:00200000:1.0:1775773260.907197:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773260.907213:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1730, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773260.907217:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1730, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773260.907241:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1730,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.907292:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.907301:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.907310:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773260.907332:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293376:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773260.907405:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a540 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.912874:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293376:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773260.912900:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912923:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912931:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912941:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912952:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912963:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912977:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.912985:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773260.912996:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c700 x1862032664293376/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773301 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773260.913006:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773260.913018:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.913028:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1513 (0 0) 00000080:00200000:1.0:1775773260.913033:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773260.913116:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x5e9:0x0](ffff98297d9a63c8) 00000080:00200000:1.0:1775773260.913235:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x5e9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773260.913245:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5e9:0x0](ffff98297d9a63c8) for lock 0x12ae90f9442ebd66 00000080:00010000:1.0:1775773260.913255:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5e9:0x0](ffff98297d9a63c8) 00000080:00010000:1.0:1775773260.913258:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebd66 from it ffffa414081d3df8 00010000:00010000:1.0:1775773260.913261:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.913268:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773260.913300:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773260.913307:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1730, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773260.913347:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 2/0,0 mode: PR/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 113 lru_type: 1 00010000:00010000:1.0:1775773260.913740:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773260.913764:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5e9:0x0]: losing layout lock 00010000:00010000:1.0:1775773260.913799:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.913806:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 1/0,0 mode: --/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 113 lru_type: 1 00010000:00010000:1.0:1775773260.913811:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1775773260.913814:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd66 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c4fac expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.913839:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a600/0x12ae90f9442ebd51 lrc: 0/0,0 mode: --/PR res: [0x2000013a2:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4f6d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 113 lru_type: 1 00000020:00100000:1.0:1775773260.913857:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773260.913865:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4c700 x1862032664293632/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773260.913888:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293632:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773260.913966:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ac00 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.917795:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293632:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773260.917892:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x5e9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773260.917906:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773260.917957:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x5e9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773260.917969:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x5e9:0x0](ffff98297d9a63c8) 00000080:00200000:1.0:1775773260.918058:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773260.918098:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1114, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773260.918103:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1114, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773260.918125:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1114,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.918176:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.918185:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.918194:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773260.918212:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293760:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773260.918289:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a540 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.922682:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293760:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773260.922708:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922735:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922748:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922759:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922771:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922784:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922792:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922798:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773260.922808:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4f100 x1862032664293760/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773301 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773260.922830:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922839:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/937 (0 0) 00000002:00010000:1.0:1775773260.922854:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x3a9:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773260.922897:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922905:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.922914:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773260.922933:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293888:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773260.923004:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a540 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.926047:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664293888:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773260.926095:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926108:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926119:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926129:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926141:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926149:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926161:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773260.926172:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4df80 x1862032664293888/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773301 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773260.926183:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926194:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/937 (0 0) 00000080:00200000:1.0:1775773260.926212:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773260.926239:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x3a9:0x0](ffff98297d9a63c8) 00000080:00010000:1.0:1775773260.926331:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x3a9:0x0](ffff98297d9a63c8) for remote lock 0x12ae90f9442ebd74 00000080:00010000:1.0:1775773260.926337:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x3a9:0x0](ffff98297d9a63c8) for lock 0x12ae90f9442ebd6d 00000080:00010000:1.0:1775773260.926344:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x3a9:0x0](ffff98297d9a63c8) 00000080:00010000:1.0:1775773260.926346:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebd6d from it ffffa414081d3df8 00010000:00010000:1.0:1775773260.926350:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926359:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773260.926372:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebd74 from it ffffa414081d3df8 00010000:00010000:1.0:1775773260.926375:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773260.926382:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773260.926399:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773260.926406:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1114, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773260.926449:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.926483:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.926490:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773260.926523:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebd6d lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4fe4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773260.926559:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773260.926571:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664294016:192.168.201.118@tcp:103:rm.0 00010000:00010000:1.0:1775773260.926655:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.926679:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773260.926686:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773260.926688:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebd74 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x3a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c4ff2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773260.926706:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773260.928764:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4f100 x1862032664294272/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773260.928838:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664294272:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773260.928893:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ac00 going to sleep for 36 seconds 00000100:00100000:1.0:1775773260.942504:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664294272:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773260.942922:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x3a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773260.942933:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773260.943156:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x3a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773260.943170:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x3a9:0x0](ffff98297d9a63c8) 00000100:00100000:1.0:1775773260.960739:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664295040:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773260.964346:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664295040:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773260.990850:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664295680:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773260.994358:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664295680:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.010975:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664296320:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.013581:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664296320:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.023151:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664296576:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.023240:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x354:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.023253:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.023308:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x354:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.023322:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x354:0x0](ffff982952593248) 00000080:00200000:1.0:1775773261.023427:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.023444:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2118, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.023451:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2118, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.023472:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2118,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.023521:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.023531:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.023555:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.023581:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664296704:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.023651:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ac00 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.027295:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664296704:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.027321:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027351:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027360:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027372:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027382:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027409:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027416:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.027429:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4df80 x1862032664296704/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.027442:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027453:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1714 (0 0) 00000002:00010000:1.0:1775773261.027469:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6b2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.027515:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027523:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.027532:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.027566:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664296832:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.027641:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ac00 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.031999:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664296832:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.032021:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032034:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032046:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032056:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032104:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032114:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032122:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.032134:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68a80 x1862032664296832/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.032147:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032173:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1714 (0 0) 00000080:00200000:1.0:1775773261.032192:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.032224:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6b2:0x0](ffff98297d9a1988) 00000080:00010000:1.0:1775773261.032318:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6b2:0x0](ffff98297d9a1988) for remote lock 0x12ae90f9442ebdb3 00000080:00010000:1.0:1775773261.032324:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6b2:0x0](ffff98297d9a1988) for lock 0x12ae90f9442ebdac 00000080:00010000:1.0:1775773261.032332:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6b2:0x0](ffff98297d9a1988) 00000080:00010000:1.0:1775773261.032336:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebdac from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.032340:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032351:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.032365:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebdb3 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.032368:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.032376:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.032394:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.032402:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2118, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.032447:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.032486:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.032492:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.032526:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdac lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c51a4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773261.032551:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773261.032565:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664296960:192.168.201.118@tcp:103:rm.0 00010000:00010000:1.0:1775773261.032656:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.032682:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.032689:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.032691:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdb3 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6b2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c51b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.032708:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.032715:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4df80 x1862032664297216/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.044223:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664297216:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.044323:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.056306:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664297216:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.056387:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6b2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.056398:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.056447:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6b2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.056459:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6b2:0x0](ffff98297d9a1988) 00000080:00200000:1.0:1775773261.056538:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.056558:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2037, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.056563:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2037, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.056582:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2037,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.056648:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.056658:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.056667:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.056685:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664297344:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.056748:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ac00 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.062783:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664297344:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.062808:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062850:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062860:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062871:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062882:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062901:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062909:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062916:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.062927:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3100 x1862032664297344/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.062938:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.062951:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.062961:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1665 (0 0) 00000080:00200000:1.0:1775773261.062968:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.062994:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x681:0x0](ffff98297d9a1988) 00000080:00200000:2.0:1775773261.063521:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x681:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.063540:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x681:0x0](ffff98297d9a1988) for lock 0x12ae90f9442ebdba 00000080:00010000:2.0:1775773261.063551:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x681:0x0](ffff98297d9a1988) 00000080:00010000:2.0:1775773261.063555:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebdba from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.063559:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.063569:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.063602:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.063612:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2037, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.063668:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.063692:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x681:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.063893:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.063901:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.063903:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ebdba lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x681:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5206 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.063925:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.063933:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7100 x1862032664297600/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.063955:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664297600:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.064032:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d180 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.122326:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bdc00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299264:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.122442:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x378:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122450:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122458:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122465:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122473:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122477:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122481:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.122488:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bdc00 x1862032664299264/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.122496:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122504:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/888 (0 0) 00000080:00200000:1.0:1775773261.122519:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.122556:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x378:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.122633:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x378:0x0](ffff98297d9a21c8) for remote lock 0x12ae90f9442ebde4 00000080:00010000:1.0:1775773261.122636:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x378:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebddd 00000080:00010000:1.0:1775773261.122642:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x378:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.122645:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebddd from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.122647:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ebddd lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5310 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122653:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ebddd lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5310 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.122662:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebde4 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.122663:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.122668:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.122681:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.122687:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.440, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.122717:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ebddd lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5310 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.122745:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ebddd lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5310 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.122749:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.122772:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166be00/0x12ae90f9442ebddd lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5310 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.122826:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.122853:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.122859:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.122862:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a400/0x12ae90f9442ebde4 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x378:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c531e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.122880:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.122888:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1c00 x1862032664299648/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.122903:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299648:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.122926:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.138417:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299648:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.138502:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x378:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.138515:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.138574:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x378:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.138588:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x378:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.138692:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.138709:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.719, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.138714:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.719, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.138748:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.719,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.138802:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.138824:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.138835:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.138865:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299776:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.138936:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.143465:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299776:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.143483:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143506:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143512:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143520:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143526:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143535:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143540:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143548:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.143556:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2d80 x1862032664299776/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.143730:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143787:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/718 (0 0) 00000002:00010000:1.0:1775773261.143805:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x2ce:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.143859:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143867:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.143875:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.143895:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299904:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.143963:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.151025:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664299904:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.151050:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151063:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151112:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151124:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151145:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151151:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.151163:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664299904/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.151176:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151188:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/718 (0 0) 00000080:00200000:1.0:1775773261.151205:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.151482:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x2ce:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.151601:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2ce:0x0](ffff98297d9a21c8) for remote lock 0x12ae90f9442ebdf2 00000080:00010000:1.0:1775773261.151607:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2ce:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebdeb 00000080:00010000:1.0:1775773261.151615:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x2ce:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.151619:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebdeb from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.151623:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151633:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.151646:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebdf2 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.151649:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.151656:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.151674:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.151682:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.719, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.151898:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.151939:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.151944:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.151961:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebdeb lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5372 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773261.151995:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664300032:192.168.201.118@tcp:103:rm.0 00010000:00010000:1.0:1775773261.152090:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.152118:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.152125:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.152128:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebdf2 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2ce:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5380 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.152151:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.152158:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2d80 x1862032664300288/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.152174:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300288:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.152199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.154700:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664300032:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.160216:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300288:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.160304:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2ce:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.160317:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.160370:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x2ce:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.160386:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x2ce:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.160466:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.160479:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2055, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.160483:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2055, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.160504:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2055,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.160556:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.160577:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.160588:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.160608:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300416:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.160669:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.162323:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300416:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.162338:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162358:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162365:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162375:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162385:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162395:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162408:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162415:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.162425:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664300416/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.162436:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162446:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1404 (0 0) 00000002:00010000:1.0:1775773261.162459:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x57c:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.162489:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162496:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.162503:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.162516:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300544:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.162568:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.164770:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300544:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.164791:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164804:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164828:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164837:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164847:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164852:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164859:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.164870:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1880 x1862032664300544/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.164881:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.164893:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1404 (0 0) 00000080:00200000:1.0:1775773261.164909:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.164945:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x57c:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.165057:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x57c:0x0](ffff98297d9a2a08) for remote lock 0x12ae90f9442ebe00 00000080:00010000:1.0:1775773261.165063:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x57c:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebdf9 00000080:00010000:1.0:1775773261.165102:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x57c:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.165105:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebdf9 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.165110:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.165120:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.165134:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe00 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.165138:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.165144:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.165165:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.165173:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2055, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.165231:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.165268:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.165275:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.165301:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebdf9 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c53d4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.165344:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.165369:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.165376:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.165378:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe00 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x57c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c53e2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.165397:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.165404:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0380 x1862032664300928/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.165419:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300928:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.165445:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.176603:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664300928:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.176691:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x57c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.176703:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.176765:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x57c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.176781:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x57c:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.176875:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.176890:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.273, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.176894:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.273, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.176913:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.273,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.176956:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.176966:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.176976:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.176997:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301056:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.177059:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.178994:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301056:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.179016:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179049:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179057:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179104:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179117:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179127:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179140:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179147:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.179159:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2300 x1862032664301056/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.179171:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179181:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/481 (0 0) 00000002:00010000:1.0:1775773261.179195:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x1e1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.179221:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179228:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.179235:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.179250:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301184:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.179297:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.180473:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301184:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.180486:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180497:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180506:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180514:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180523:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180530:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180537:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.180546:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3800 x1862032664301184/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.180556:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180564:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/481 (0 0) 00000080:00200000:1.0:1775773261.180579:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.180603:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x1e1:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.180695:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x1e1:0x0](ffff98297d9a21c8) for remote lock 0x12ae90f9442ebe0e 00000080:00010000:1.0:1775773261.180700:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x1e1:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebe07 00000080:00010000:1.0:1775773261.180707:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x1e1:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.180709:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe07 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.180713:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180721:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.180731:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe0e from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.180734:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.180741:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.180756:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.180764:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.273, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.180805:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.180850:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.180857:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.180886:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe07 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5436 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.180911:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.180930:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.180936:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.180939:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe0e lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x1e1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5444 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.181100:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.181107:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2300 x1862032664301568/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.181122:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301568:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.181179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.188268:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301568:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.188351:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x1e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.188365:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.188421:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x1e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.188436:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x1e1:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.188524:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.188558:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.188563:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.188585:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2279,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.188636:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.188647:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.188656:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.188677:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301696:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.188834:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.190606:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301696:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.190624:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190650:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190658:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190670:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190681:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190691:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190704:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190712:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.190722:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2a00 x1862032664301696/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.190733:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.190744:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190754:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1516 (0 0) 00000080:00200000:1.0:1775773261.190759:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.190785:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x5ec:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.190901:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x5ec:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.190912:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x5ec:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebe15 00000080:00010000:1.0:1775773261.190922:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x5ec:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.190926:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe15 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.190929:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.190938:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.190968:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.190974:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.191026:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.191050:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x5ec:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.191120:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.191128:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.191130:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe15 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x5ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5498 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.191149:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.191155:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2a00 x1862032664301952/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.191173:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301952:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.191230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.193244:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664301952:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.193305:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x5ec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.193317:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.193360:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x5ec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.193373:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x5ec:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.193457:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.193474:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.638, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.193479:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.638, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.193496:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.638,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.193563:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.193572:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.193582:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.193597:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302080:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.193694:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.195291:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302080:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.195304:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195323:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195328:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195336:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195351:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195365:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195373:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195380:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.195390:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0e00 x1862032664302080/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.195401:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.195412:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195421:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/990 (0 0) 00000080:00200000:1.0:1775773261.195425:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.195456:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x3de:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.195561:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x3de:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.195573:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x3de:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebe1c 00000080:00010000:1.0:1775773261.195583:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x3de:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.195586:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe1c from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.195588:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.195598:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.195617:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.195621:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.638, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.195666:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.195687:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x3de:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.195718:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.195723:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.195724:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe1c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x3de:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c54d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.195741:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.195747:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0e00 x1862032664302336/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.195763:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302336:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.195829:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.200521:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302336:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.200600:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x3de:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.200613:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.200672:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x3de:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.200690:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x3de:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.200795:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.200825:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1073, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.200830:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1073, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.200858:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1073,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.200908:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.200918:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.200928:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.200948:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302464:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.201020:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.203104:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302464:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.203119:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203150:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203169:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203180:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203196:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203208:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203216:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203223:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.203233:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0a80 x1862032664302464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.203243:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.203254:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203264:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1177 (0 0) 00000080:00200000:1.0:1775773261.203269:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.203292:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x499:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.203493:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x499:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.203505:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x499:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebe23 00000080:00010000:1.0:1775773261.203514:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x499:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.203518:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe23 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.203521:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.203530:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.203553:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.203559:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1073, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.203600:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.203621:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x499:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.203653:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.203660:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.203662:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe23 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x499:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5508 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.203681:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.203687:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0a80 x1862032664302720/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.203705:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302720:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.203770:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.205528:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302720:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.205689:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x499:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.205701:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.205745:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x499:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.205758:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x499:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.205851:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.205866:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1589, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.205871:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1589, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.205889:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1589,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.205933:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.205942:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.205951:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.205968:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302848:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.206019:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.207760:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302848:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.207775:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207799:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207807:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207834:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207844:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207860:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207867:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207874:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.207885:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1500 x1862032664302848/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.207896:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207904:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1438 (0 0) 00000002:00010000:1.0:1775773261.207915:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x59e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.207937:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207942:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.207947:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.207957:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302976:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.208006:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.209421:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664302976:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.209433:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209441:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209448:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209455:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209461:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209465:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209470:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.209490:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0000 x1862032664302976/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.209501:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209510:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1438 (0 0) 00000080:00200000:1.0:1775773261.209523:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.209564:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x59e:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.209660:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x59e:0x0](ffff98297d9a2a08) for remote lock 0x12ae90f9442ebe31 00000080:00010000:1.0:1775773261.209664:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x59e:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebe2a 00000080:00010000:1.0:1775773261.209670:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x59e:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.209672:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe2a from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.209675:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209681:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.209689:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe31 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.209691:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.209696:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.209707:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.209714:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1589, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.209754:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.209785:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.209791:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.209822:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe2a lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5540 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.209845:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.209868:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.209876:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.209880:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe31 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x59e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c554e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.209898:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.209905:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1500 x1862032664303360/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.209920:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303360:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.209950:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.216353:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303360:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.216423:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x59e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.216435:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.216490:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x59e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.216502:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x59e:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.216586:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.216601:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2437, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.216606:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2437, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.216627:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2437,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.216680:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.216692:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.216702:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.216721:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303488:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.216774:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.219122:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303488:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.219136:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219153:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219160:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219224:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219233:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219241:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219246:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219250:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.219257:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0700 x1862032664303488/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.219265:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219271:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1599 (0 0) 00000002:00010000:1.0:1775773261.219282:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x63f:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.219305:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219310:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.219315:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.219327:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303616:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.219443:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.220716:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664303616:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.220727:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220735:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220743:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220753:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220762:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220768:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220775:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.220789:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1180 x1862032664303616/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.220800:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220808:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1599 (0 0) 00000080:00200000:1.0:1775773261.220833:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.220857:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x63f:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.220927:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x63f:0x0](ffff98297d9a3a88) for remote lock 0x12ae90f9442ebe3f 00000080:00010000:1.0:1775773261.220930:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x63f:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebe38 00000080:00010000:1.0:1775773261.220936:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x63f:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.220938:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe38 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.220940:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220947:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.220954:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe3f from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.220956:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.220960:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.220971:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.220976:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2437, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.221008:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.221038:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.221044:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.221301:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebe38 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c55a2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.221320:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.221336:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.221498:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.221500:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe3f lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x63f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c55b0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.221518:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.221524:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0700 x1862032664304000/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.221546:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304000:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.221572:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.228145:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304000:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.228267:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x63f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.228277:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.228312:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x63f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.228321:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x63f:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.228525:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.228585:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.966, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.228590:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.966, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.228612:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.966,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.228662:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.228671:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.228678:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.228694:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304128:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.228754:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.230327:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304128:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.230348:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230365:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230371:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230382:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230390:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230399:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230407:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230414:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.230424:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2680 x1862032664304128/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.230436:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230447:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/863 (0 0) 00000002:00010000:1.0:1775773261.230459:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x35f:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.230492:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230499:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.230507:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.230523:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304256:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.231433:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.232755:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304256:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.232767:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232775:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232782:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232788:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232795:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232800:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232804:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.232831:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1f80 x1862032664304256/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.232841:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.232848:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/863 (0 0) 00000080:00200000:1.0:1775773261.232863:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.232895:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x35f:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.232984:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x35f:0x0](ffff98297d9a2a08) for remote lock 0x12ae90f9442ebe4d 00000080:00010000:1.0:1775773261.232988:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x35f:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebe46 00000080:00010000:1.0:1775773261.232993:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x35f:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.232996:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe46 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.232999:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.233005:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.233013:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe4d from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.233015:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.233020:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.233032:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.233039:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.966, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.233115:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.233143:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.233147:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.233167:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe46 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5604 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.233184:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.233196:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.233200:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.233201:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe4d lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x35f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5612 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.233212:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.233217:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2680 x1862032664304640/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.233227:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304640:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.233263:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.243283:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304640:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.243359:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x35f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.243372:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.243415:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x35f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.243425:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x35f:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.243501:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.243515:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2196, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.243518:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2196, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.243538:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2196,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.243593:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.243600:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.243607:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.243622:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304768:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.243674:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.244926:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664304768:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.244937:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244953:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244958:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244969:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244976:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244982:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244987:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.244992:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.244998:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3b80 x1862032664304768/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.245004:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.245011:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.245017:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1746 (0 0) 00000080:00200000:1.0:1775773261.245020:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.245045:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6d2:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.245135:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x6d2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.245142:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6d2:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebe54 00000080:00010000:1.0:1775773261.245149:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6d2:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.245151:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe54 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.245153:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.245159:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.245175:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.245181:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2196, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.245216:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.245232:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x6d2:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.245259:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.245265:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.245268:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe54 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6d2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5666 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.245286:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.245292:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe3b80 x1862032664305024/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.245306:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305024:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.245344:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.246795:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305024:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.246855:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6d2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.246865:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.246900:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6d2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.246910:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6d2:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.246979:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.246993:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2256, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.246998:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2256, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.247011:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2256,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.247042:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.247051:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.247060:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.247103:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305152:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.247152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.250100:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305152:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.250120:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250147:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250156:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250167:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250178:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250198:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250206:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.250217:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1f80 x1862032664305152/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.250229:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.250241:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250252:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1774 (0 0) 00000080:00200000:1.0:1775773261.250257:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.250282:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6ee:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.250393:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x6ee:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.250406:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6ee:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebe5b 00000080:00010000:1.0:1775773261.250415:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6ee:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.250419:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe5b from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.250423:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.250432:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.250457:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.250465:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2256, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.250516:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.250538:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x6ee:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.250575:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.250582:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.250584:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe5b lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c569e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.250603:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.250609:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1f80 x1862032664305408/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.250638:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305408:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.250695:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.253931:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305408:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.253995:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6ee:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.254007:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.254050:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6ee:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.254083:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6ee:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.254165:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.254181:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.962, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.254186:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.962, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.254214:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.962,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.254257:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.254267:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.254277:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.254293:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305536:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.254348:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.255560:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305536:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.255573:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255594:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255602:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255613:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255623:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255632:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255640:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255647:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.255656:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1180 x1862032664305536/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.255668:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255676:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1125 (0 0) 00000002:00010000:1.0:1775773261.255686:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x465:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.255717:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x465:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255724:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x465:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.255732:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x465:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.255744:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305664:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.255779:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.257219:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664305664:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.257232:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x465:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257243:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257252:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257262:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257272:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257280:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257287:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.257301:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0000 x1862032664305664/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.257311:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257320:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1125 (0 0) 00000080:00200000:1.0:1775773261.257332:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.257357:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x465:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.257433:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x465:0x0](ffff98297d9a3a88) for remote lock 0x12ae90f9442ebe69 00000080:00010000:1.0:1775773261.257437:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x465:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebe62 00000080:00010000:1.0:1775773261.257445:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x465:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.257448:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe62 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.257451:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257460:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.257479:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebe69 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.257482:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.257490:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.257506:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.257512:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.962, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.257563:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.257592:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.257598:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.257613:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe62 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c56d6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.257632:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.257648:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.257654:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.257657:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe69 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x465:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c56e4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.257670:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.257676:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1180 x1862032664306048/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.257690:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306048:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.257736:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.262927:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306048:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.262998:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x465:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.263010:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.263095:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x465:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.263113:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x465:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.263187:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.263198:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1267, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.263202:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1267, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.263219:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1267,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.263258:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.263265:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.263272:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.263285:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306176:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.263339:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.264547:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306176:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.264559:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264580:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264587:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264601:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264610:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264620:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264627:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264635:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.264645:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3800 x1862032664306176/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.264655:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.264665:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264674:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1292 (0 0) 00000080:00200000:1.0:1775773261.264678:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.264699:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x50c:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.264917:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x50c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.264929:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x50c:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebe70 00000080:00010000:1.0:1775773261.264938:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x50c:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.264942:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebe70 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.264945:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.264954:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.264978:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.264985:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1267, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.265024:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.265045:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x50c:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.265137:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.265145:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.265147:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebe70 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x50c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5738 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.265168:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.265176:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe3800 x1862032664306432/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.265194:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306432:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.265249:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.268046:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306432:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.268147:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x50c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.268160:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.268215:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x50c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.268229:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x50c:0x0](ffff98297d9a21c8) 00000080:00200000:1.0:1775773261.268314:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.268330:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2288, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.268334:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2288, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.268371:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2288,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.268419:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.268430:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.268441:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.268461:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306560:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.268528:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.272397:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306560:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.272421:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272451:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272460:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272472:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272482:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272493:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272501:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272507:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.272518:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1880 x1862032664306560/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.272543:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ebe77 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5770 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272554:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1791 (0 0) 00000002:00010000:1.0:1775773261.272579:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6ff:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.272619:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272635:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.272645:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.272663:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306688:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.272744:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.276059:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664306688:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.276106:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276118:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276133:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276144:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276153:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276158:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276163:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.276174:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3480 x1862032664306688/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.276183:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.276192:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1791 (0 0) 00000080:00200000:1.0:1775773261.276207:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1775773261.276784:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebe7e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6ff:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c577e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773261.294516:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664307840:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.296887:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664307840:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.307619:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773261.307633:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664308480:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.308872:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664308480:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.325902:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bf800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664309120:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.325981:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6a4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.325993:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.326047:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6a4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.326061:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6a4:0x0](ffff982952597448) 00000080:00200000:1.0:1775773261.326235:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.326251:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.581, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.326256:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.581, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.326280:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.581,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.326328:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.326338:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.326348:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.326368:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664309248:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.326440:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.334963:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664309248:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.334988:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335019:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335027:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335039:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335050:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335063:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335090:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335097:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.335108:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1880 x1862032664309248/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.335120:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335130:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/643 (0 0) 00000002:00010000:1.0:1775773261.335146:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x283:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.335190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x283:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335197:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x283:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.335206:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x283:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.335227:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664309376:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.335310:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a780 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.337357:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664309376:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.337373:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x283:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337385:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337394:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337404:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337414:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337422:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337429:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.337440:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3100 x1862032664309376/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.337451:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337461:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/643 (0 0) 00000080:00200000:1.0:1775773261.337477:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.337507:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x283:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.337602:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x283:0x0](ffff98297d9a21c8) for remote lock 0x12ae90f9442ebeb6 00000080:00010000:1.0:1775773261.337608:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x283:0x0](ffff98297d9a21c8) for lock 0x12ae90f9442ebeaf 00000080:00010000:1.0:1775773261.337616:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x283:0x0](ffff98297d9a21c8) 00000080:00010000:1.0:1775773261.337619:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebeaf from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.337623:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337633:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.337644:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebeb6 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.337647:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.337653:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeb6 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5914 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.337671:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.337678:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.581, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.337721:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.337756:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.337763:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.337786:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebeaf lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x283:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5906 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773261.337832:0:404329:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775773261.337845:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664309504:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.346372:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664309504:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.364235:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664310528:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.368538:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664310528:192.168.201.118@tcp:103:rm.0 00010000:00010000:1.0:1775773261.384948:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442ebed9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x72e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5a10 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.384982:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442ebed9 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x72e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a10 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.384990:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.384993:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166b200/0x12ae90f9442ebed9 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x72e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a10 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.385011:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.385018:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe3100 x1862032664311424/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.385034:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311424:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.385091:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.391127:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311424:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.391360:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x72e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.391372:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.391427:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x72e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.391442:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x72e:0x0](ffff982952597448) 00000080:00200000:1.0:1775773261.391535:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.391552:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2150, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.391557:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2150, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.391580:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2150,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.391638:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.391649:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.391658:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.391678:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311552:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.391752:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.393553:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311552:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.393572:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393597:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393605:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393616:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393628:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393638:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393646:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393653:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.393664:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe1880 x1862032664311552/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.393676:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393686:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1724 (0 0) 00000002:00010000:1.0:1775773261.393698:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6bc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.393731:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393738:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.393745:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.393765:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311680:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.393838:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.395430:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664311680:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.395447:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395457:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395466:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395476:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395493:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395501:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395507:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.395517:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3480 x1862032664311680/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.395534:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395544:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1724 (0 0) 00000080:00200000:1.0:1775773261.395560:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.395587:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6bc:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.395683:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6bc:0x0](ffff98297d9a3a88) for remote lock 0x12ae90f9442ebee7 00000080:00010000:1.0:1775773261.395688:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6bc:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebee0 00000080:00010000:1.0:1775773261.395695:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6bc:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.395699:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebee0 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.395702:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395711:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.395722:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebee7 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.395725:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.395732:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.395749:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.395757:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2150, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.395800:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.395849:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.395856:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.395884:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebee0 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.395906:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.395929:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.395937:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.395939:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebee7 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6bc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5a72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.395958:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.395967:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe1880 x1862032664312064/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.395983:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312064:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.396053:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.404847:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312064:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.404927:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6bc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.404940:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.404992:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6bc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.405008:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6bc:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.405117:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.405133:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1494, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.405138:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1494, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.405161:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1494,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.405223:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.405233:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.405243:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.405264:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312192:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.405328:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.407125:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312192:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.407140:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407162:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407170:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407181:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407204:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407217:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407225:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407232:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.407242:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0000 x1862032664312192/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.407250:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.407259:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407267:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1390 (0 0) 00000080:00200000:1.0:1775773261.407272:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.407301:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x56e:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.407405:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x56e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.407417:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x56e:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebeee 00000080:00010000:1.0:1775773261.407426:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x56e:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.407430:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebeee from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.407433:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.407441:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.407464:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.407469:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1494, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.407503:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.407525:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x56e:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.407566:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.407573:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.407576:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebeee lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x56e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5ac6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.407593:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.407600:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0000 x1862032664312448/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.407617:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312448:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.407662:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.409925:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312448:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.410004:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x56e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.410016:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.410111:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x56e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.410127:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x56e:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.410213:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.410230:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2236, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.410235:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2236, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.410254:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2236,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.410300:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.410309:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.410319:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.410337:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312576:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.410408:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.416042:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312576:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.416088:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416118:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416127:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416144:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416155:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416168:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416176:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416183:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.416194:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3480 x1862032664312576/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.416205:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.416218:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416228:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1494 (0 0) 00000080:00200000:1.0:1775773261.416234:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.416268:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x5d6:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.416393:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x5d6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.416405:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x5d6:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebef5 00000080:00010000:1.0:1775773261.416414:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x5d6:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.416418:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebef5 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.416421:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.416430:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.416501:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.416509:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2236, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.416592:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.416617:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x5d6:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.416658:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.416665:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.416668:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebef5 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x5d6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5afe expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.416688:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.416695:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe3480 x1862032664312832/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.416715:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312832:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.416785:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.419857:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312832:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.419930:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x5d6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.419941:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.419992:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x5d6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.420004:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x5d6:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.420115:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.420130:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2098, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.420134:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2098, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.420154:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2098,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.420203:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.420213:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.420223:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.420241:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312960:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.420308:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.422321:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982970b68a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664312960:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.422338:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422363:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422371:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422382:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422393:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422403:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422411:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422417:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.422428:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982970b68a80 x1862032664312960/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.422439:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422448:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1703 (0 0) 00000002:00010000:1.0:1775773261.422461:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.422513:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422521:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.422529:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.422544:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313088:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.422644:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.424683:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313088:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.424697:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424717:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424725:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424735:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424746:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424753:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424760:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.424776:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5c00 x1862032664313088/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.424788:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424797:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1703 (0 0) 00000080:00200000:1.0:1775773261.424824:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.424852:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6a7:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.424945:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6a7:0x0](ffff98297d9a2a08) for remote lock 0x12ae90f9442ebf03 00000080:00010000:1.0:1775773261.424950:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6a7:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebefc 00000080:00010000:1.0:1775773261.424958:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6a7:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.424961:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebefc from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.424965:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424975:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.424986:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebf03 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.424990:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.424997:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.425016:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.425023:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2098, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.425132:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.425169:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.425176:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.425192:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebefc lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b36 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.425214:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.425249:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.425255:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.425257:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf03 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6a7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b44 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.425273:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.425279:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c6d80 x1862032664313472/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.425295:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313472:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.425518:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.434587:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313472:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.434678:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.434691:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.434743:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.434758:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6a7:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.434928:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.434945:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1088, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.434950:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1088, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.434972:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1088,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.435037:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.435048:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.435057:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.435459:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313600:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.435530:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.439389:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313600:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.439417:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439445:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439454:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439470:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439481:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439492:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439508:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439516:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.439528:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4380 x1862032664313600/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.439538:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439546:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/926 (0 0) 00000002:00010000:1.0:1775773261.439559:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x39e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.439598:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439606:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.439626:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.439642:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313728:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.439696:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.440825:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664313728:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.440836:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440842:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440849:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440856:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440865:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440872:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440878:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.440897:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5880 x1862032664313728/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.440906:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.440916:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/926 (0 0) 00000080:00200000:1.0:1775773261.440930:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.440960:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x39e:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.441045:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x39e:0x0](ffff98297d9a3a88) for remote lock 0x12ae90f9442ebf11 00000080:00010000:1.0:1775773261.441049:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x39e:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebf0a 00000080:00010000:1.0:1775773261.441055:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x39e:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.441058:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf0a from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.441060:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.441113:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.441122:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebf11 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.441124:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.441129:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.441145:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.441151:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1088, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.441189:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.441222:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.441228:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.441252:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf0a lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5b98 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.441275:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.441296:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.441302:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.441304:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf11 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x39e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5ba6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.441318:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.441323:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4380 x1862032664314112/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.441334:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314112:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.441384:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.447273:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314112:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.447355:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x39e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.447366:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.447412:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x39e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.447426:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x39e:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.447510:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.447526:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1403, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.447531:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1403, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.447560:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1403,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.447612:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.447622:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.447632:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.447651:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314240:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.447715:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.449754:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314240:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.449769:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449792:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449800:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449824:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449833:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449843:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449850:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449867:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.449877:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c6300 x1862032664314240/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.449886:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.449896:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.449905:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1353 (0 0) 00000080:00200000:1.0:1775773261.449909:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.449932:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x549:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.450028:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x549:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.450039:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x549:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebf18 00000080:00010000:1.0:1775773261.450047:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x549:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.450050:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf18 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.450054:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.450062:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.450133:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.450137:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1403, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.450168:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.450187:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x549:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.450211:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.450216:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.450218:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf18 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x549:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5bfa expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.450230:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.450234:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c6300 x1862032664314496/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.450245:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314496:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.450287:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.452519:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314496:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.452579:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x549:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.452590:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.452635:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x549:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.452644:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x549:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.452707:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.452721:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1903, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.452726:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1903, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.452743:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1903,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.452781:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.452788:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.452795:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.452815:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314624:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.452963:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.456309:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314624:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.456326:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456345:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456351:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456360:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456367:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456375:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456380:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456384:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.456390:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664314624/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.456397:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.456405:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456411:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1332 (0 0) 00000080:00200000:1.0:1775773261.456415:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.456438:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x534:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.456541:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x534:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.456553:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x534:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebf1f 00000080:00010000:1.0:1775773261.456562:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x534:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.456566:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf1f from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.456569:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.456578:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.456604:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.456611:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1903, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.456653:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.456668:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x534:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.456697:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.456703:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.456705:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf1f lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x534:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5c32 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.456724:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.456730:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c7800 x1862032664314880/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.456750:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314880:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.456816:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.460164:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664314880:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.460324:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x534:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.460343:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.460381:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x534:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.460392:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x534:0x0](ffff98297d9a3a88) 00000080:00200000:1.0:1775773261.460457:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.460470:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1690, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.460474:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1690, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.460493:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1690,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.460541:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.460550:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.460561:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.460579:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315008:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.460642:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a9c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.462869:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315008:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.462889:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462916:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462925:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462936:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462947:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462959:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462964:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462969:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.462976:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c6a00 x1862032664315008/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.462983:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.462991:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.462997:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1220 (0 0) 00000080:00200000:1.0:1775773261.463001:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.463022:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x4c4:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.463914:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x4c4:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.463927:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4c4:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebf26 00000080:00010000:1.0:1775773261.463937:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x4c4:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.463940:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf26 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.463944:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.463954:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.463980:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.463986:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1690, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.464030:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.464051:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x4c4:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.464148:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.464156:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.464159:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf26 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4c4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5c6a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.464178:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.464185:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c6a00 x1862032664315264/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.464205:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315264:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.464307:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.466939:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315264:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.467010:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x4c4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.467023:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.467107:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x4c4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.467120:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x4c4:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.467208:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.467222:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.680, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.467226:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.680, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.467244:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.680,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.467287:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf2d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.467297:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf2d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.467306:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7800/0x12ae90f9442ebf2d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.467324:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315392:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.472057:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315648:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.472135:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2b8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.472146:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.472180:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x2b8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.472189:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x2b8:0x0](ffff982952595b88) 00000080:00200000:1.0:1775773261.472264:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.472282:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.731, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.472286:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.731, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.472304:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.731,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.472338:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.472347:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.472355:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.472371:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315776:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.472418:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.475528:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315776:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.475549:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475575:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475583:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475594:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475603:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475614:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475623:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475631:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.475642:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4a80 x1862032664315776/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.475653:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475663:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/725 (0 0) 00000002:00010000:1.0:1775773261.475689:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x2d5:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.475727:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475735:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.475744:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.475760:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315904:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.475969:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ab40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.478134:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664315904:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.478153:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478166:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478177:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478186:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478197:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478204:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478211:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.478221:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5500 x1862032664315904/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.478234:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478245:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/725 (0 0) 00000080:00200000:1.0:1775773261.478261:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.478290:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x2d5:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.478392:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2d5:0x0](ffff98297d9a3a88) for remote lock 0x12ae90f9442ebf3b 00000080:00010000:1.0:1775773261.478398:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2d5:0x0](ffff98297d9a3a88) for lock 0x12ae90f9442ebf34 00000080:00010000:1.0:1775773261.478406:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x2d5:0x0](ffff98297d9a3a88) 00000080:00010000:1.0:1775773261.478410:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf34 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.478414:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478424:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.478435:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebf3b from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.478438:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.478445:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.478463:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.478471:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.731, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.478541:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.478578:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.478585:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.478612:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf34 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5cda expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.478639:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.478659:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.478665:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.478668:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf3b lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5ce8 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.478684:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.478690:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4a80 x1862032664316288/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.478822:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664316288:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.479204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ae40 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.485482:0:404824:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862017593038720 00000100:00100000:2.0:1775773261.485502:0:404824:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862017593038720 00000100:00100000:2.0:1775773261.485918:0:404824:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.118@tcp, seq: 295 00000100:00100000:2.0:1775773261.485930:0:404824:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff982981c34000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:14dd209b-d605-4024-a2b7-27ec435c0914+5:201909:x1862017593038720:12345-192.168.201.118@tcp:104: 00010000:00010000:2.0:1775773261.485942:0:404824:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x12ae90f9442ebf34 - lock disappeared 00010000:00010000:2.0:1775773261.485995:0:404824:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.201.118@tcp lock=0x12ae90f9442ebf34: rc = 0 req@ffff982981c34000 x1862017593038720/t0(0) o104->14dd209b-d605-4024-a2b7-27ec435c0914@192.168.201.118@tcp:437/0 lens 328/224 e 0 to 0 dl 1775773272 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.486008:0:404824:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff982981c34000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:14dd209b-d605-4024-a2b7-27ec435c0914+5:201909:x1862017593038720:12345-192.168.201.118@tcp:104: Request processed in 81us (594us total) trans 0 rc -22/-22 00000100:00100000:2.0:1775773261.486018:0:404824:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.118@tcp, seq: 295 00000100:00100000:1.0:1775773261.489990:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664316288:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.490056:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2d5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.490150:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00010000:00010000:1.0:1775773261.511327:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442ebf50 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3f9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5d82 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.511362:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442ebf50 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x3f9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5d82 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.511370:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.511373:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166a000/0x12ae90f9442ebf50 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x3f9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5d82 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.511397:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.511404:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4000 x1862032664317312/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.511422:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317312:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.511450:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ae40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.518892:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317312:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.518957:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x3f9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.518968:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.519005:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x3f9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.519013:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x3f9:0x0](ffff982952595b88) 00000080:00200000:1.0:1775773261.519104:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.519120:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2072, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.519124:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2072, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.519141:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2072,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.519183:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.519191:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.519198:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.519212:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317440:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.519271:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ae40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.520802:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317440:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.520834:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520858:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520867:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520878:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520888:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520899:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520907:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520914:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.520926:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4700 x1862032664317440/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.520938:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.520948:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1687 (0 0) 00000002:00010000:1.0:1775773261.520961:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x697:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.521000:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x697:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.521007:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x697:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.521015:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x697:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.521030:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317568:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.521207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ae40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.522632:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317568:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.522706:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x697:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522717:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522727:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522735:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522742:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522749:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522756:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.522766:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5180 x1862032664317568/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.522777:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522787:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1687 (0 0) 00000080:00200000:1.0:1775773261.522804:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.522851:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x697:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.522935:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x697:0x0](ffff98297d9a3248) for remote lock 0x12ae90f9442ebf5e 00000080:00010000:1.0:1775773261.522939:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x697:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebf57 00000080:00010000:1.0:1775773261.522945:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x697:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.522948:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf57 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.522950:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522958:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.522970:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebf5e from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.522973:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.522981:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.522998:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.523005:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2072, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.523045:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.523165:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.523174:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.523419:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ebf57 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5dd6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.523455:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.523481:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.523493:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.523495:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf5e lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x697:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5de4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.523513:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.523519:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4700 x1862032664317952/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.523536:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317952:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.523575:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a6c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.531797:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664317952:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.531880:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x697:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.531891:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.531933:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x697:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.531946:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x697:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.532035:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.532049:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1762, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.532053:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1762, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.532093:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1762,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.532137:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.532151:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.532159:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.532174:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664318080:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.532241:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54ae40 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.536536:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664318080:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.536560:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536595:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536603:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536614:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536625:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536637:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536645:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536653:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.536664:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c6680 x1862032664318080/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.536676:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.536693:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536704:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1526 (0 0) 00000080:00200000:1.0:1775773261.536711:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.536739:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x5f6:0x0](ffff98297d9a2a08) 00000080:00200000:1.0:1775773261.536879:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x5f6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.536892:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5f6:0x0](ffff98297d9a2a08) for lock 0x12ae90f9442ebf65 00000080:00010000:1.0:1775773261.536901:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5f6:0x0](ffff98297d9a2a08) 00000080:00010000:1.0:1775773261.536904:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf65 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.536908:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.536917:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.536947:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.536955:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1762, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.537001:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.537023:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5f6:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.537105:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.537112:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.537115:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf65 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5f6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5e38 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.537135:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.537142:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c6680 x1862032664318336/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.537575:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664318336:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.538654:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a6c0 going to sleep for 36 seconds 00010000:00010000:1.0:1775773261.560645:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf73 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x204:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c5e7e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.560674:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf73 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x204:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5e7e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.560681:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.560684:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf73 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x204:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c5e7e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.560705:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.560713:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5f80 x1862032664318976/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.560729:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664318976:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.560775:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.572862:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664318976:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.572966:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x204:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.572988:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.573042:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x204:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.573058:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x204:0x0](ffff982952593248) 00000080:00200000:1.0:1775773261.573269:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.573357:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2304, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.573362:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2304, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.573384:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2304,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.573439:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.573450:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.573461:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.573482:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319104:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.573597:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.576870:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319104:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.576889:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576911:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576917:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576924:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576932:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576940:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576945:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576950:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.576959:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7b80 x1862032664319104/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.576969:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.576980:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.576990:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1799 (0 0) 00000080:00200000:1.0:1775773261.576996:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.577029:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x707:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.577165:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x707:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.577174:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x707:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebf7a 00000080:00010000:1.0:1775773261.577275:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x707:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.577280:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf7a from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.577284:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.577293:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.577320:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.577328:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2304, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.577368:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.577383:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x707:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.577411:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.577416:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.577418:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf7a lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x707:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5ed2 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.577432:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.577436:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c7b80 x1862032664319360/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.577451:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319360:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.577522:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.583240:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319360:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.583322:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x707:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.583335:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.583379:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x707:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.583389:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x707:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.583467:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.583483:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.889, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.583488:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.889, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.583510:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.889,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.583669:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.583680:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.583691:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.583779:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319488:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.583852:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.585540:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319488:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.585559:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585584:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585591:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585606:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585616:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585626:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585633:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585640:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.585650:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7100 x1862032664319488/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.585660:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.585671:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585680:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/816 (0 0) 00000080:00200000:1.0:1775773261.585685:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.585711:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x330:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773261.585831:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x330:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.585844:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x330:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ebf81 00000080:00010000:1.0:1775773261.585853:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x330:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773261.585856:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf81 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.585859:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.585867:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.585891:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.585898:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.889, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.585949:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.585969:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x330:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.586001:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.586007:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.586009:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebf81 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x330:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5f0a expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.586027:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.586033:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c7100 x1862032664319744/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.586050:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319744:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.586149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.590261:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319744:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.590341:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x330:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.590353:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.590401:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x330:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.590416:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x330:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773261.590520:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.590549:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2056, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.590554:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2056, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.590574:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2056,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.590630:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.590650:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.590660:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.590679:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319872:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.590740:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.593744:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664319872:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.593763:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593785:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593790:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593800:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593820:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593833:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593840:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593847:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.593859:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7480 x1862032664319872/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.593870:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.593881:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.593888:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1674 (0 0) 00000080:00200000:1.0:1775773261.593893:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.593920:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x68a:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.594024:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x68a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.594032:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x68a:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebf88 00000080:00010000:1.0:1775773261.594039:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x68a:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.594042:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf88 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.594044:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.594050:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.594097:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.594106:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2056, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.594157:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.594172:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x68a:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.594202:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.594206:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.594208:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebf88 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x68a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5f42 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.594220:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.594225:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c7480 x1862032664320128/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.594240:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664320128:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.594299:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.602768:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664320128:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.602861:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x68a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.602874:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.602928:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x68a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.603346:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x68a:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.603439:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.603454:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1628, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.603459:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1628, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000100:00100000:1.0:1775773261.615062:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664320512:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.625960:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664320512:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773261.649114:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664320896:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.649136:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649162:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649170:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649181:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649190:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649207:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649215:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649222:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.649231:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982965ec2300 x1862032664320896/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.649243:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.649254:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649263:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/775 (0 0) 00000080:00200000:1.0:1775773261.649269:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.649295:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x307:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.649410:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x307:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.649421:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x307:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebf9d 00000080:00010000:1.0:1775773261.649430:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x307:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.649434:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebf9d from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.649437:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.649451:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.649477:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.649548:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.189, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.649592:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.649613:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x307:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.649649:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.649655:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.649658:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff98294166bc00/0x12ae90f9442ebf9d lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x307:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c5fdc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.649676:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.649682:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5180 x1862032664321152/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.649700:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321152:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.649765:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.655480:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321152:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.655560:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x307:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.655571:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.655620:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x307:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.655636:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x307:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.655730:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.655750:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.835, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.655754:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.835, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.655775:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.835,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.655838:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.655848:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.655859:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.655879:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321280:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.655982:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.670183:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321280:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.670205:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670235:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670244:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670256:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670273:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670286:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670295:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670303:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.670313:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5500 x1862032664321280/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.670325:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670345:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/788 (0 0) 00000002:00010000:1.0:1775773261.670361:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x314:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.670421:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x314:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670429:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x314:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.670439:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x314:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.670462:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321408:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.670522:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.673368:0:412860:0:(client.c:3082:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff982942b6f800: committing for last_committed 8589971191 gen 1 00000100:00100000:1.0:1775773261.673701:0:412860:0:(client.c:3131:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff982977d4d880 x1862032664293248/t8589971154(8589971154) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 608/696 e 0 to 0 dl 1775773301 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1775773261.673779:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321408:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.673814:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x314:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673825:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673837:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673847:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673860:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673868:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673876:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.673886:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4e00 x1862032664321408/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.673898:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.673910:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/788 (0 0) 00000080:00200000:1.0:1775773261.673922:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.673962:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x314:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.674056:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x314:0x0](ffff98297d9a3248) for remote lock 0x12ae90f9442ebfab 00000080:00010000:1.0:1775773261.674062:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x314:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebfa4 00000080:00010000:1.0:1775773261.674759:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x314:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.674763:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfa4 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.674766:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.674778:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.674791:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebfab from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.674794:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.674801:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.674838:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.674846:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.835, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.674898:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.675001:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.675009:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.675356:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfa4 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6022 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.675394:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.675418:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.675425:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.675427:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfab lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x314:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6030 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.675444:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.675451:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5500 x1862032664321792/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.675467:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321792:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.675503:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.690679:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321792:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.690759:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x314:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.690771:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.690832:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x314:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.690846:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x314:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.690945:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.690963:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.751, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.690967:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.751, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.690989:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.751,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.691039:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.691050:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.691060:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.691297:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321920:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.691559:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.693641:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664321920:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.693661:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693752:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693763:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693779:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693791:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693817:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693826:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693833:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.693845:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7100 x1862032664321920/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.693855:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.693866:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.693877:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1039 (0 0) 00000080:00200000:1.0:1775773261.693883:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.693914:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x40f:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773261.694040:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x40f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.694053:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x40f:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ebfb2 00000080:00010000:1.0:1775773261.694095:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x40f:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773261.694100:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfb2 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.694104:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.694114:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.694143:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.694150:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.751, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.694195:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.694217:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x40f:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.694253:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.694261:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.694264:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfb2 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6084 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.694283:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.694290:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c7100 x1862032664322176/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.694308:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322176:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.694381:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.696902:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322176:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.696985:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x40f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.696998:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.697050:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x40f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.697063:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x40f:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773261.697175:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.697191:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1269, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.697196:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1269, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.697217:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1269,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.697263:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.697273:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.697283:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.697301:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322304:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.697377:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.704053:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322304:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.704092:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704125:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704134:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704146:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704158:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704170:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704179:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704187:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.704198:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5f80 x1862032664322304/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.704209:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.704221:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704231:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1293 (0 0) 00000080:00200000:1.0:1775773261.704250:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.704280:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x50d:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.704394:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x50d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.704406:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x50d:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebfb9 00000080:00010000:1.0:1775773261.704415:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x50d:0x0](ffff98297d9a3248) 00000080:00010000:1.0:1775773261.704419:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfb9 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.704422:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.704432:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.704460:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.704468:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1269, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.704510:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.704531:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x50d:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.704569:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.704575:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.704578:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfb9 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x50d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c60bc expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.704597:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.704603:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5f80 x1862032664322560/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.704624:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322560:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.704682:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.712789:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322560:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.712879:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x50d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.712891:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.712951:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x50d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.712965:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x50d:0x0](ffff98297d9a3248) 00000080:00200000:1.0:1775773261.713051:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.717615:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.887, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.717625:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.887, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.717652:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.887,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.717709:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.717720:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.717731:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.717754:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322688:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.717900:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.719748:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322688:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.719766:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719792:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719801:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719826:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719837:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719849:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719856:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719864:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.719875:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4a80 x1862032664322688/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.719886:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.719899:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.719909:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1094 (0 0) 00000080:00200000:1.0:1775773261.719914:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.719956:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x446:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773261.720167:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x446:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773261.720181:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x446:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ebfc0 00000080:00010000:1.0:1775773261.720191:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x446:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773261.720195:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfc0 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.720198:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.720207:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.720236:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.720244:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.887, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.720284:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.720305:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x446:0x0]: losing layout lock 00010000:00010000:1.0:1775773261.720337:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.720344:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.720347:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc0 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c60f4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.720365:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.720372:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4a80 x1862032664322944/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.720389:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322944:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.720444:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.726596:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664322944:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.726672:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x446:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.726684:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.726730:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x446:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.726743:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x446:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773261.726847:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.726863:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1442, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.726868:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1442, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.726887:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1442,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.726945:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.726956:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.726965:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.726983:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323072:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.727048:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.731023:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323072:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.731046:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731106:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731116:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731128:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731139:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731152:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731160:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731168:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.731180:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c6a00 x1862032664323072/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.731193:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731204:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1098 (0 0) 00000002:00010000:1.0:1775773261.731218:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x44a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.731258:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731266:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.731276:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.731296:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323200:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.731364:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.735272:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323200:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.735290:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735299:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735308:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735315:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735329:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735334:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735339:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.735349:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664323200/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.735358:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735367:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1098 (0 0) 00000080:00200000:1.0:1775773261.735382:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.735407:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x44a:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773261.735487:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x44a:0x0](ffff98297d9a7448) for remote lock 0x12ae90f9442ebfce 00000080:00010000:1.0:1775773261.735491:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x44a:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ebfc7 00000080:00010000:1.0:1775773261.735496:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x44a:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773261.735499:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfc7 from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.735501:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735508:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773261.735517:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebfce from it ffffa414081d3df8 00010000:00010000:1.0:1775773261.735518:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.735523:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773261.735536:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773261.735543:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1442, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773261.735575:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.735602:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.735606:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.735620:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ebfc7 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c612c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.735636:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.735647:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773261.735651:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773261.735652:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfce lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x44a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c613a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773261.735662:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773261.735666:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c6a00 x1862032664323584/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773261.735677:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323584:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773261.735735:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.749436:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323584:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773261.749520:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x44a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773261.749533:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773261.749585:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x44a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773261.749599:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x44a:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773261.749685:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773261.749701:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1994, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773261.749705:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1994, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773261.749727:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1994,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773261.750190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.750202:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.750214:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773261.750234:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323712:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773261.750307:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773261.752846:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323712:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773261.752867:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752893:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752901:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752918:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752928:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752940:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752948:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752955:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773261.752965:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4380 x1862032664323712/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773261.752976:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773261.752986:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773261.752995:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1644 (0 0) 00000080:00200000:1.0:1775773261.753002:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773261.753033:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x66c:0x0](ffff98297d9a3248) 00000080:00200000:2.0:1775773261.753286:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x66c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.753302:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x66c:0x0](ffff98297d9a3248) for lock 0x12ae90f9442ebfd5 00000080:00010000:2.0:1775773261.753313:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x66c:0x0](ffff98297d9a3248) 00000080:00010000:2.0:1775773261.753317:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfd5 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.753321:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.753343:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.753373:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.753380:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1994, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.753423:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.753444:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x66c:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.753483:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.753491:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.753494:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6600/0x12ae90f9442ebfd5 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x66c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c618e expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.753514:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.753522:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7480 x1862032664323968/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.753548:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323968:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.753607:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.756973:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664323968:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.757094:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x66c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.757107:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.758465:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x66c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.758487:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x66c:0x0](ffff98297d9a3248) 00000080:00200000:2.0:1775773261.758585:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.758601:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.292, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.758606:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.292, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.758626:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.292,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.758678:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.758688:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.758699:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.758723:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324096:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.758823:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.767691:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324096:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.767715:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767750:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767759:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767776:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767788:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767812:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767821:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767829:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.767842:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4000 x1862032664324096/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.767863:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773261.767881:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.767892:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/823 (0 0) 00000080:00200000:2.0:1775773261.767898:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.767940:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x337:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.768056:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x337:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.768094:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x337:0x0](ffff98296faeba88) for lock 0x12ae90f9442ebfdc 00000080:00010000:2.0:1775773261.768104:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x337:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.768108:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfdc from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.768112:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.768122:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.768149:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.768157:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.292, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.768210:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.768230:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x337:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.768266:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.768274:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.768277:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfdc lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x337:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c61c6 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.768296:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.768302:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4000 x1862032664324352/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.768323:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324352:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.768382:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.773738:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324352:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.773843:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x337:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.773855:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.773936:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x337:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.773949:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x337:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.774031:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.774046:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1004, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.774051:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1004, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.774217:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1004,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.774270:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.774280:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.774289:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.774309:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324480:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.774478:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.777560:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324480:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.777583:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777609:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777618:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777629:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777640:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777652:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777660:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777668:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.777675:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5c00 x1862032664324480/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.777684:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777691:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/884 (0 0) 00000002:00010000:2.0:1775773261.777765:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x374:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.777798:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x374:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777816:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x374:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.777822:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x374:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.777837:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324608:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.777889:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.779426:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324608:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.779440:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x374:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779446:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779461:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779470:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779485:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779493:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779500:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.779510:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6a00 x1862032664324608/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.779521:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779817:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/884 (0 0) 00000080:00200000:2.0:1775773261.779833:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.779864:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x374:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.779959:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x374:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ebfea 00000080:00010000:2.0:1775773261.779965:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x374:0x0](ffff98296fae9148) for lock 0x12ae90f9442ebfe3 00000080:00010000:2.0:1775773261.779972:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x374:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.779975:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfe3 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.779978:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.779988:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.780000:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ebfea from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.780004:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.780012:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.780030:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.780037:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1004, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.780308:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.780344:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.780351:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.780378:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebfe3 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c61fe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773261.780403:0:404332:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1775773261.780417:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664324736:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773261.780499:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.780523:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.782788:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.782793:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfea lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x374:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c620c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.782835:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.782848:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5c00 x1862032664324992/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.782870:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324992:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.782938:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.793112:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664324992:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.793187:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x374:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.793198:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.793242:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x374:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.793251:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x374:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.793331:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.793348:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.351, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.793352:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.351, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.793371:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.351,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.793418:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.793426:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.793433:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.793449:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325120:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.793554:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.795572:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325120:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.795593:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795620:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795628:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795646:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795658:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795669:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795678:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795685:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.795700:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4e00 x1862032664325120/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.795709:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773261.795717:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795724:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/524 (0 0) 00000080:00200000:2.0:1775773261.795729:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.795756:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x20c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.795866:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x20c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.795878:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x20c:0x0](ffff98296faeba88) for lock 0x12ae90f9442ebff1 00000080:00010000:2.0:1775773261.795888:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x20c:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.795891:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebff1 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.795894:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.795903:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.795929:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.795937:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.351, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.795983:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.796010:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x20c:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.796041:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.796047:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.796049:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebff1 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x20c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6260 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.796102:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.796109:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4e00 x1862032664325376/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.796129:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325376:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.796196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.800481:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325376:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.800652:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x20c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.800665:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.800713:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x20c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.800726:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x20c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.800815:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.800833:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2379, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.800838:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2379, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.800861:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2379,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.800913:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.800925:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.800932:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.800947:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325504:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.801010:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.805162:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325504:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.805186:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805214:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805222:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805233:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805243:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805256:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805269:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805277:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.805288:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6d80 x1862032664325504/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.805299:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773261.805311:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805320:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1572 (0 0) 00000080:00200000:2.0:1775773261.805327:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.805357:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x624:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.805489:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x624:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.805501:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x624:0x0](ffff98296fae9148) for lock 0x12ae90f9442ebff8 00000080:00010000:2.0:1775773261.805511:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x624:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.805515:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebff8 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.805518:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.805541:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.805575:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.805583:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2379, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.805643:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.805669:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x624:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.805710:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.805718:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.805720:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ebff8 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x624:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6298 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.805742:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.805749:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6d80 x1862032664325760/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.805771:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325760:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.805857:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.808604:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325760:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.808676:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x624:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.808688:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.809902:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x624:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.809916:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x624:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.810012:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.810026:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.520, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.810030:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.520, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.810047:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.520,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.810110:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.810118:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.810125:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.810141:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325888:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.810202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.816612:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664325888:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.816639:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816668:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816676:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816687:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816699:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816725:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816734:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816741:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.816752:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5500 x1862032664325888/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.816763:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773261.816775:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.816785:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/936 (0 0) 00000080:00200000:2.0:1775773261.816792:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.816841:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x3a8:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.816979:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x3a8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.816991:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x3a8:0x0](ffff98296faeba88) for lock 0x12ae90f9442ebfff 00000080:00010000:2.0:1775773261.817002:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x3a8:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.817006:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ebfff from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.817009:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.817023:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.817050:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.817057:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.520, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.817695:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.817792:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x3a8:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.817849:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.817856:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.817859:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ebfff lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x3a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c62d0 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.817881:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.817889:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5500 x1862032664326144/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.817910:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326144:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.818262:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.825339:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326144:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.825421:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x3a8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.825433:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.825479:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x3a8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.825493:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x3a8:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.825579:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.825595:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.660, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.825609:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.660, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.825630:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.660,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.825672:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.825679:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.825686:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.825701:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326272:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.825767:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.830566:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326272:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.830590:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830630:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830638:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830649:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830667:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830680:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830689:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830696:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.830708:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6a00 x1862032664326272/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.830729:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830739:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/681 (0 0) 00000002:00010000:2.0:1775773261.830755:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x2a9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.830812:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830820:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.830829:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.830851:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326400:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.830913:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.832723:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326400:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.832740:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832751:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832761:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832773:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832785:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832792:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832810:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.832821:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6300 x1862032664326400/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.832840:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.832851:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/681 (0 0) 00000080:00200000:2.0:1775773261.832868:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.832898:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x2a9:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.832991:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2a9:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec00d 00000080:00010000:2.0:1775773261.832997:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2a9:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec006 00000080:00010000:2.0:1775773261.833004:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x2a9:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.833008:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec006 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.833011:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.833020:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.833033:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec00d from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.833036:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.833043:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.833061:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.833099:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.660, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.833147:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.833183:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.833190:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.833213:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec006 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6308 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.833239:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.833260:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.833266:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.833268:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec00d lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2a9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6316 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.833285:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.833291:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6a00 x1862032664326784/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.833309:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326784:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.833357:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.838028:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664326528:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.847557:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326784:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.847635:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.847647:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.847693:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x2a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.847705:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x2a9:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.847787:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.847813:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.42, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.847818:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.42, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.847842:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.42,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.847893:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.847902:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.847911:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.847931:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326912:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.848001:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.852861:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664326912:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.852887:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852914:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852922:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852939:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852949:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852961:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852968:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.852976:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.852986:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b6300 x1862032664326912/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.852997:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.853005:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/355 (0 0) 00000002:00010000:2.0:1775773261.853021:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x163:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.853061:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x163:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.853111:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x163:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.853121:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x163:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.853148:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327040:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.853213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.859462:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327040:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.859485:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x163:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859498:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859511:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859523:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859535:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859543:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859550:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.859566:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4a80 x1862032664327040/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.859580:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859593:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/355 (0 0) 00000080:00200000:2.0:1775773261.859610:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.859644:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x163:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.859742:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x163:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec01b 00000080:00010000:2.0:1775773261.859747:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x163:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec014 00000080:00010000:2.0:1775773261.859755:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x163:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.859759:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec014 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.859763:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859772:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.859786:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec01b from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.859789:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.859796:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.859831:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.859838:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.42, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.859884:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.859923:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.859930:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.859958:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec014 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c636a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.859999:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.860024:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.860031:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.860034:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec01b lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x163:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6378 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.860053:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.860061:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b6300 x1862032664327424/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.860122:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327424:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.860162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.862755:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664327168:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.870400:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b6300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327424:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.870476:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x163:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.870487:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.870531:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x163:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.870543:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x163:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.870633:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.870648:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2285, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.870652:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2285, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.870673:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2285,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.870739:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.870748:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.870758:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.870776:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327552:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.870857:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.872978:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327552:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.872991:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873010:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873016:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873025:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873032:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873040:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873045:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873049:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.873055:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4a80 x1862032664327552/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.873063:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873092:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1790 (0 0) 00000002:00010000:2.0:1775773261.873103:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6fe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.873130:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873135:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.873141:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.873167:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327680:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.873224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.877164:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664327680:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.877186:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877197:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877207:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877218:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877230:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877237:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877244:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.877259:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4700 x1862032664327680/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.877270:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877282:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1790 (0 0) 00000080:00200000:2.0:1775773261.877298:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.877333:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6fe:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.877416:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6fe:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec029 00000080:00010000:2.0:1775773261.877421:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6fe:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec022 00000080:00010000:2.0:1775773261.877428:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6fe:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.877431:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec022 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.877435:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877443:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.877455:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec029 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.877457:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.877463:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.877480:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.877488:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2285, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.877532:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.877572:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.877577:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.877598:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec022 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c63cc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773261.877619:0:404331:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1775773261.877835:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664327808:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773261.877905:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.877924:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.877928:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.877929:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec029 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c63da expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.877942:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.877946:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4a80 x1862032664328064/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.877957:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328064:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.877974:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.879861:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664327808:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.884997:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328064:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.885475:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6fe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.885488:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.885559:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6fe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.885574:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6fe:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.885670:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.885687:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.797, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.885691:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.797, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.885712:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.797,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.885761:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.885771:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.885781:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.885811:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328192:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.886279:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.889782:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328192:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.889815:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889838:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889848:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889857:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889865:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889875:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889880:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889885:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.889892:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b4700 x1862032664328192/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.889901:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889906:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/766 (0 0) 00000002:00010000:2.0:1775773261.889919:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x2fe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.889952:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889958:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.889963:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.889977:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328320:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.890149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.891667:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328320:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.891679:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891686:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891700:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891706:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891713:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891717:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891725:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.891731:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5880 x1862032664328320/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.891738:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891745:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/766 (0 0) 00000080:00200000:2.0:1775773261.891757:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.891789:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x2fe:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.891880:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2fe:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec037 00000080:00010000:2.0:1775773261.891884:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2fe:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec030 00000080:00010000:2.0:1775773261.891891:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x2fe:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.891893:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec030 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.891895:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891902:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.891911:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec037 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.891913:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.891918:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.891931:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.891937:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.797, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.891969:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.891993:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.892006:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.892031:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec030 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c642e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.892054:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.892098:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.892102:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.892104:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec037 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2fe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c643c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.892118:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.892124:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b4700 x1862032664328704/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.892135:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328704:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.892175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.893604:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664328448:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.897185:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b4700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328704:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.897279:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2fe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.897292:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.897347:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x2fe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.897360:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x2fe:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.897467:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.897484:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2093, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.897489:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2093, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.897512:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2093,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.897561:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.897571:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.897581:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.897600:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328832:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.897669:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.899549:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328832:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.899569:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899602:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899610:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899623:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899635:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899647:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899656:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899663:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.899673:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5880 x1862032664328832/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.899686:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899696:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1415 (0 0) 00000002:00010000:2.0:1775773261.899710:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x587:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.899741:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x587:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899749:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x587:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.899757:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x587:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.899772:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328960:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.899838:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.900845:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664328960:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.900858:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x587:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900868:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900877:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900886:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900895:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900902:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900909:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.900926:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7b80 x1862032664328960/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.900938:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.900947:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1415 (0 0) 00000080:00200000:2.0:1775773261.900963:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.900994:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x587:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.901108:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x587:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec045 00000080:00010000:2.0:1775773261.901113:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x587:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec03e 00000080:00010000:2.0:1775773261.901121:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x587:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.901125:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec03e from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.901128:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.901137:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.901146:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec045 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.901148:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.901152:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.901166:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.901172:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2093, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.901205:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.901229:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.901242:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.901270:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec03e lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6490 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773261.901293:0:404331:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1775773261.901305:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664329088:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773261.901363:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.901379:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.901383:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.901385:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec045 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x587:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c649e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.901396:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.901400:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5880 x1862032664329344/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.901417:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329344:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.901438:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.905649:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664329088:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.910943:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329344:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.911023:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x587:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.911037:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.911134:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x587:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.911148:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x587:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.911240:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.911258:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2133, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.911263:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2133, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.911287:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2133,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.911337:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.911346:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.911356:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.911375:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329472:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.911440:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.913879:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329472:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.913901:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913930:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913939:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913950:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913961:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913973:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913980:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.913987:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.913997:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b7b80 x1862032664329472/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.914008:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.914017:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1719 (0 0) 00000002:00010000:2.0:1775773261.914033:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x6b7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.914093:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.914100:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.914109:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.914127:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329600:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.914196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.919049:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329600:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.919099:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919112:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919123:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919134:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919147:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919156:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919162:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.919172:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032664329600/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.919185:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919198:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1719 (0 0) 00000080:00200000:2.0:1775773261.919216:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.919252:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6b7:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.919365:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6b7:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec053 00000080:00010000:2.0:1775773261.919369:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6b7:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec04c 00000080:00010000:2.0:1775773261.919376:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6b7:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.919379:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec04c from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.919381:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919389:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.919399:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec053 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.919402:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.919409:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.919429:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.919437:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2133, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.919501:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.919540:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.919545:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.919578:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec04c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c64f2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.919604:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.919643:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.919651:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.919655:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec053 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6b7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6500 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.919673:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.919680:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b7b80 x1862032664329984/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.919697:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329984:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.919730:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.922670:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664329728:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.928879:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b7b80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664329984:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.928979:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6b7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.928991:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.929041:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6b7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.929054:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6b7:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.929197:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.929213:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.664, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.929218:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.664, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.929240:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.664,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.929294:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.929302:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.929312:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.929343:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330112:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.929413:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.932124:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330112:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.932146:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932181:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932189:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932200:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932211:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932224:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932233:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932241:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.932252:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032664330112/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.932265:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932274:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/684 (0 0) 00000002:00010000:2.0:1775773261.932289:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x2ac:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.932334:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932342:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.932350:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.932369:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330240:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.932453:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.936400:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330240:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.936422:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936434:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936445:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936456:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936468:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936487:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936495:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.936507:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bd500 x1862032664330240/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.936520:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936532:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/684 (0 0) 00000080:00200000:2.0:1775773261.936550:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.936581:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x2ac:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.936675:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2ac:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec061 00000080:00010000:2.0:1775773261.936681:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x2ac:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec05a 00000080:00010000:2.0:1775773261.936689:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x2ac:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.936692:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec05a from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.936696:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936706:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.936719:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec061 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.936722:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.936729:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.936748:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.936756:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.664, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.936813:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.936850:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.936856:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.936884:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec05a lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6554 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.936935:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.936963:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.936969:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.936972:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec061 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x2ac:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6562 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.937002:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.937009:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bd880 x1862032664330624/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.937025:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330624:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.937054:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.938861:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664330368:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.949344:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330624:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.949422:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x2ac:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.949433:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.949471:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x2ac:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.949480:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x2ac:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.949559:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.949572:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1532, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.949575:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1532, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.949593:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1532,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.949642:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.949652:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.949663:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.949683:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330752:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.949754:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.952450:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330752:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.952473:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952507:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952516:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952527:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952538:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952550:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952559:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952566:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.952577:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bd500 x1862032664330752/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.952590:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952600:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1143 (0 0) 00000002:00010000:2.0:1775773261.952616:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x477:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.952656:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x477:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952663:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x477:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.952671:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x477:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.952690:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330880:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.952763:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.957461:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664330880:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.957484:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x477:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957497:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957508:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957518:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957530:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957539:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957546:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.957558:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bd180 x1862032664330880/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.957571:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957583:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1143 (0 0) 00000080:00200000:2.0:1775773261.957601:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.957631:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x477:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.957738:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x477:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec06f 00000080:00010000:2.0:1775773261.957744:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x477:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec068 00000080:00010000:2.0:1775773261.957751:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x477:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.957754:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec068 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.957758:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957767:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.957780:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec06f from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.957783:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.957806:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.957827:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.957840:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1532, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.957888:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.957928:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.957935:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.957969:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec068 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c65b6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773261.957996:0:404332:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1775773261.958011:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664331008:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773261.958135:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.958160:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.958167:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.958169:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec06f lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x477:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c65c4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.958205:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.958212:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bd500 x1862032664331264/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.958235:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331264:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.958261:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.959624:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664331008:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.967780:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331264:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.967879:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x477:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.967892:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.967942:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x477:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.967955:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x477:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.968044:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.968061:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1081, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.968128:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1081, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.968151:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1081,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.968213:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.968224:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.968235:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.968256:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331392:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.968341:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.969686:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331392:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.969701:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969725:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969732:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969743:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969754:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969764:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969772:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969778:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.969789:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bd180 x1862032664331392/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.969812:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773261.969829:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.969838:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1182 (0 0) 00000080:00200000:2.0:1775773261.969843:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.969871:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x49e:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.969970:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x49e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773261.969981:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x49e:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec076 00000080:00010000:2.0:1775773261.969990:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x49e:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.969993:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec076 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.969997:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.970005:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.970028:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.970035:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1081, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.970108:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.970132:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x49e:0x0]: losing layout lock 00010000:00010000:2.0:1775773261.970168:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.970174:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.970178:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec076 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x49e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6618 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.970205:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.970214:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bd180 x1862032664331648/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.970232:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331648:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.970289:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.971970:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bd180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331648:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.972031:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x49e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.972041:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.972128:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x49e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.972148:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x49e:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773261.972247:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.972262:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.932, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.972266:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.932, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.972286:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.932,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.972340:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.972350:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.972361:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.972378:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bc000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331776:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.972442:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.978204:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bc000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331776:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.978228:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978257:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978266:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978276:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978287:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978299:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978307:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978313:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.978324:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bc000 x1862032664331776/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.978335:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978343:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1108 (0 0) 00000002:00010000:2.0:1775773261.978358:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x454:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.978400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x454:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978408:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x454:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.978416:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x454:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.978436:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331904:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.978501:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.979997:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664331904:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.980013:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x454:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980024:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980034:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980045:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980056:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980093:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980102:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.980112:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bce00 x1862032664331904/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.980125:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980134:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1108 (0 0) 00000080:00200000:2.0:1775773261.980150:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.980179:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x454:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.980286:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x454:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec084 00000080:00010000:2.0:1775773261.980291:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x454:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec07d 00000080:00010000:2.0:1775773261.980300:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x454:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773261.980303:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec07d from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.980307:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980315:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.980342:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec084 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.980345:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.980353:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.980371:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.980378:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.932, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.980420:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.980452:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.980467:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.980495:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec07d lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6650 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.980518:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.980539:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.980543:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.980544:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec084 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x454:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c665e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.980558:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.983004:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bc000 x1862032664332288/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.983046:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bc000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332288:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.983131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.985121:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664332032:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773261.990987:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bc000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332288:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773261.991061:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x454:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773261.991110:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773261.991176:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x454:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773261.991190:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x454:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773261.991282:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773261.991301:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.973, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773261.991305:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.973, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773261.991334:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.973,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.991381:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.991391:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.991401:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.991420:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332416:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.991482:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.993494:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332416:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.993514:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993541:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993549:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993559:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993570:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993591:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993598:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993606:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.993617:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bce00 x1862032664332416/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.993629:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993640:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/868 (0 0) 00000002:00010000:2.0:1775773261.993654:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x364:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773261.993682:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x364:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993687:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x364:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.993693:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x364:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773261.993704:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332544:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773261.993756:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773261.996061:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332544:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773261.996097:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x364:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996108:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996119:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996129:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996141:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996149:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996156:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773261.996167:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bca80 x1862032664332544/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773302 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773261.996179:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996189:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/868 (0 0) 00000080:00200000:2.0:1775773261.996205:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773261.996234:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x364:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.996328:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x364:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec092 00000080:00010000:2.0:1775773261.996334:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x364:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec08b 00000080:00010000:2.0:1775773261.996341:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x364:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773261.996344:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec08b from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.996348:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996357:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773261.996370:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec092 from it ffffa414081d3df8 00010000:00010000:2.0:1775773261.996373:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773261.996379:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773261.996397:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773261.996405:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.973, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773261.996449:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.996483:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.996490:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.996519:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec08b lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c66b2 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.996558:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.996683:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773261.996691:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773261.996694:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec092 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x364:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c66c0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773261.996715:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773261.996722:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bce00 x1862032664332928/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773261.996737:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332928:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773261.996770:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.002315:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664332672:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.004911:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664332928:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.005238:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x364:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.005250:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.005303:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x364:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.005314:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x364:0x0](ffff98296fae9148) 00000080:00200000:2.0:1775773262.005403:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.005416:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.005420:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.005440:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.279,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.005483:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.005489:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.005497:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.005511:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333056:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.005605:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.009442:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333056:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.009468:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009499:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009508:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009519:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009530:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009548:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009557:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009564:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.009576:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829705bca80 x1862032664333056/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.009588:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773262.009602:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009612:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/818 (0 0) 00000080:00200000:2.0:1775773262.009619:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.009652:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x332:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.009775:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x332:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773262.009787:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x332:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec099 00000080:00010000:2.0:1775773262.009815:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x332:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.009819:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec099 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.009823:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.009832:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.009867:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.009876:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.279, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.009923:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.009947:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x332:0x0]: losing layout lock 00010000:00010000:2.0:1775773262.010003:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.010016:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.010019:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec099 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x332:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6714 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.010039:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.010057:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829705bca80 x1862032664333312/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.010112:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333312:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.010198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.013157:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829705bca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333312:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.013229:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x332:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.013252:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.013300:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x332:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.013312:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x332:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.013397:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.013411:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2186, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.013415:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2186, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.013437:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2186,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.013487:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec0a0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.013497:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec0a0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.013505:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec0a0 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.013524:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333440:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.013585:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.018029:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664333440:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.018163:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7600/0x12ae90f9442ec0a0 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c674c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.040680:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664334336:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.045402:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664334336:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.058407:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664335360:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.061040:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664335360:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.070804:0:404329:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775773262.070816:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664336000:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.072489:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664336000:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.094291:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773262.094307:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664337408:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.095356:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664337408:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.103244:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338048:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.103305:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x3a5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.103316:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.103358:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x3a5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.103372:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x3a5:0x0](ffff982952593248) 00000080:00200000:2.0:1775773262.103452:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.103465:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.103469:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.103487:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1874,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.103528:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.103538:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.103547:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.103562:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338176:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.103626:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.105002:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338176:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.105016:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105050:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105059:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105095:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105106:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105117:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105125:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105133:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.105142:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829793b5180 x1862032664338176/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.105152:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773262.105164:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105174:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1316 (0 0) 00000080:00200000:2.0:1775773262.105178:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.105214:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x524:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.105306:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x524:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773262.105317:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x524:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec102 00000080:00010000:2.0:1775773262.105325:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x524:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.105329:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec102 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.105332:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.105340:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.105361:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.105367:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.105404:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.105424:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x524:0x0]: losing layout lock 00010000:00010000:2.0:1775773262.105454:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.105461:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.105464:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec102 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x524:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6a16 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.105482:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.105488:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829793b5180 x1862032664338432/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.105504:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338432:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.105549:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.110873:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829793b5180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338432:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.110957:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x524:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.110972:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.111025:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x524:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.111038:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x524:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.111144:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.111161:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1948, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.111166:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1948, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.111189:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1948,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.111240:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.111250:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.111260:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.111279:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338560:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.111350:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.116978:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338560:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.117002:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117031:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117041:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117051:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117062:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117128:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117134:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117138:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.117146:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94a80 x1862032664338560/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.117157:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117167:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1619 (0 0) 00000002:00010000:2.0:1775773262.117183:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x653:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.117224:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x653:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117229:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x653:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.117238:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x653:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.117258:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338688:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.117332:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.119768:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664338688:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.119806:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x653:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119817:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119829:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119840:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119853:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119860:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119867:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.119878:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f96d80 x1862032664338688/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.119890:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.119902:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1619 (0 0) 00000080:00200000:2.0:1775773262.119921:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.119962:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x653:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.120107:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x653:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec110 00000080:00010000:2.0:1775773262.120112:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x653:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec109 00000080:00010000:2.0:1775773262.120122:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x653:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.120126:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec109 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.120130:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.120139:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.120164:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec110 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.120168:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.120176:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.120196:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.120204:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1948, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.120263:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.120304:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.120310:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.120340:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec109 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6a4e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.120383:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.120410:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.120417:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.120419:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec110 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x653:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6a5c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.120439:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.120446:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f94a80 x1862032664339072/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.120463:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339072:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.120496:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.121724:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664338816:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.129468:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339072:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.129555:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x653:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.129567:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.129622:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x653:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.129634:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x653:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.129721:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.129737:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1006, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.129741:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1006, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.129763:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1006,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.129828:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.129838:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.129848:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.129868:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339200:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.129946:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.132831:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339200:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.133109:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133149:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133160:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133172:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133182:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133192:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133210:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133218:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.133229:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f96d80 x1862032664339200/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.133244:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133255:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1144 (0 0) 00000002:00010000:2.0:1775773262.133268:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x478:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.133309:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x478:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133317:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x478:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.133326:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x478:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.133344:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339328:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.133413:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.135949:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339328:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.135988:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x478:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.135999:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136009:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136019:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136036:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136042:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136048:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.136058:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f97100 x1862032664339328/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.136089:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136100:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1144 (0 0) 00000080:00200000:2.0:1775773262.136124:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.136150:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x478:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.136245:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x478:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec11e 00000080:00010000:2.0:1775773262.136250:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x478:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec117 00000080:00010000:2.0:1775773262.136258:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x478:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.136268:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec117 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.136272:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136281:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.136293:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec11e from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.136296:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.136303:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.136321:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.136328:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1006, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.136382:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.136420:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.136427:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.136452:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7000/0x12ae90f9442ec117 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ab0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773262.136473:0:404332:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1775773262.136484:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664339456:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773262.136544:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.136564:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.136571:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.136573:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec11e lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x478:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6abe expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.136588:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.136603:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f96d80 x1862032664339712/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.136619:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339712:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.136648:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.138047:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664339456:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.144189:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339712:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.144280:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x478:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.144292:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.144332:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x478:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.144341:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x478:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.144411:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.144424:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.12, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.144427:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.12, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.144445:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.12,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.144495:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.144506:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.144521:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.144540:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339840:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.144604:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.145775:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339840:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.145790:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145826:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145835:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145846:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145857:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145867:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145874:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145885:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.145894:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f97100 x1862032664339840/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.145905:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145914:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/691 (0 0) 00000002:00010000:2.0:1775773262.145926:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x2b3:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.145951:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145957:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.145964:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.145977:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339968:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.146057:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d600 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.147758:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664339968:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.147771:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147778:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147786:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147792:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147812:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147822:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147828:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.147838:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94e00 x1862032664339968/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.147850:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.147861:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/691 (0 0) 00000080:00200000:2.0:1775773262.147880:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.147914:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x2b3:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773262.147998:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x2b3:0x0](ffff98296fae9148) for remote lock 0x12ae90f9442ec12c 00000080:00010000:2.0:1775773262.148003:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x2b3:0x0](ffff98296fae9148) for lock 0x12ae90f9442ec125 00000080:00010000:2.0:1775773262.148011:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x2b3:0x0](ffff98296fae9148) 00000080:00010000:2.0:1775773262.148016:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec125 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.148020:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.148029:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.148040:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec12c from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.148044:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.148050:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.148097:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.148105:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.12, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.148152:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.148189:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.148196:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.148220:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7e00/0x12ae90f9442ec125 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6b12 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.148245:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.148288:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.148296:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.148299:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7200/0x12ae90f9442ec12c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x2b3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6b20 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.148319:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.148326:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f97100 x1862032664340352/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.148343:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664340352:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.148372:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d3c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.150049:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664340096:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.158384:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664340352:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.180840:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664341120:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.183385:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664341120:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.193395:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664341760:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.195341:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664341760:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.211715:0:404329:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775773262.211727:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664342400:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.219914:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664342400:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.233274:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773262.233293:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664343040:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.237129:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664343040:192.168.201.118@tcp:103:rm.0 00000080:00200000:1.0:1775773262.242247:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x51f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.242266:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x51f:0x0](ffff982952595b88) 00000080:00200000:1.0:1775773262.242358:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.242373:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1636, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.242378:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1636, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.242471:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1636,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.242521:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.242542:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.242553:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.242573:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343424:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.242642:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.247152:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343424:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.247177:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247205:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247213:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247224:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247235:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247247:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247255:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247262:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.247272:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5880 x1862032664343424/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.247283:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247292:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1194 (0 0) 00000002:00010000:1.0:1775773262.247306:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x4aa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.247363:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247370:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.247380:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.247399:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343552:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.247465:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.248842:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343552:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.248856:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248865:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248874:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248885:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248895:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248903:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248910:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.248923:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5c00 x1862032664343552/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.248932:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.248941:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1194 (0 0) 00000080:00200000:1.0:1775773262.248956:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.248997:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x4aa:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.249117:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4aa:0x0](ffff98297d9a7448) for remote lock 0x12ae90f9442ec179 00000080:00010000:1.0:1775773262.249123:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4aa:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ec172 00000080:00010000:1.0:1775773262.249134:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x4aa:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.249138:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec172 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.249141:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.249154:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.249166:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec179 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.249169:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.249175:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.249195:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.249203:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1636, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.249244:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.249274:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.249280:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.249306:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec172 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6d34 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.249360:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.249386:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.249392:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.249395:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec179 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4aa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6d42 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.249412:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.249419:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5880 x1862032664343936/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.249434:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343936:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.249463:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.255314:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5880 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664343936:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.255389:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x4aa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.255400:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.255451:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x4aa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.255463:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x4aa:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773262.255538:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.255556:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1803, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.255560:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1803, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.255579:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1803,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.255624:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.255633:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.255643:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.255659:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344064:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.255717:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.257259:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344064:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.257275:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257300:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257308:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257323:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257335:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257346:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257354:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257361:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.257369:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c5c00 x1862032664344064/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.257376:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.257384:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257392:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1276 (0 0) 00000080:00200000:1.0:1775773262.257397:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.257422:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x4fc:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.257510:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x4fc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.257518:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4fc:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec180 00000080:00010000:1.0:1775773262.257525:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x4fc:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.257536:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec180 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.257543:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.257551:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.257578:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.257585:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1803, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.257626:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.257647:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x4fc:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.257681:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.257688:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.257691:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec180 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6d96 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.257709:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.257715:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c5c00 x1862032664344320/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.257733:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344320:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.257782:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.260426:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5c00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344320:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.260493:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x4fc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.260505:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.260559:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x4fc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.260576:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x4fc:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.260656:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.260670:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1822, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.260675:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1822, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.260693:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1822,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.260735:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.260744:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.260753:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.260768:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344448:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.260875:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.264436:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344448:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.264468:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264499:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264508:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264521:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264532:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264543:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264552:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264559:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.264572:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4380 x1862032664344448/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.264586:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264595:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1286 (0 0) 00000002:00010000:1.0:1775773262.264611:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x506:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.264653:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x506:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264660:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x506:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.264668:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x506:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.264686:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344576:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.264753:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.267694:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344576:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.267806:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x506:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267818:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267829:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267840:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267852:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267859:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267867:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.267882:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664344576/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.267895:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.267906:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1286 (0 0) 00000080:00200000:1.0:1775773262.267925:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.267959:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x506:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.268057:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x506:0x0](ffff98297d9a7448) for remote lock 0x12ae90f9442ec18e 00000080:00010000:1.0:1775773262.268090:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x506:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ec187 00000080:00010000:1.0:1775773262.268098:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x506:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.268101:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec187 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.268105:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.268114:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.268129:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec18e from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.268132:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.268139:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.268161:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.268170:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1822, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.268217:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.268255:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.268263:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.268290:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec187 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6dce expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.268315:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.268340:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.268348:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.268350:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec18e lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x506:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ddc expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.268370:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.268377:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4380 x1862032664344960/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.268395:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344960:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.268427:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.276707:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664344960:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.276789:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x506:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.276815:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.276866:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x506:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.276881:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x506:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773262.276966:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.276981:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.107, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.276985:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.107, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.277008:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.107,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.277061:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.277101:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.277112:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.277133:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345088:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.277252:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.278861:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345088:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.278880:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278904:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278912:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278928:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278937:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278959:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278967:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.278974:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.278985:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c4e00 x1862032664345088/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.278997:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.279008:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/741 (0 0) 00000002:00010000:1.0:1775773262.279022:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x2e5:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.279097:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.279104:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.279113:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.279129:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345216:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.279182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.288059:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345216:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.288119:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288133:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288144:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288154:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288166:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288174:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288181:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.288196:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664345216/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.288208:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288220:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/741 (0 0) 00000080:00200000:1.0:1775773262.288544:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.288574:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x2e5:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.288703:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x2e5:0x0](ffff98297d9a7448) for remote lock 0x12ae90f9442ec19c 00000080:00010000:1.0:1775773262.288709:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x2e5:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ec195 00000080:00010000:1.0:1775773262.288728:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x2e5:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.288732:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec195 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.288735:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288742:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.288751:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec19c from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.288752:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.288757:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.288771:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.288778:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.107, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.288837:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.288875:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.288882:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.289003:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec195 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e30 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.289024:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.289043:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.289048:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.289049:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec19c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x2e5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e3e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.289062:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.289141:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9829745c4e00 x1862032664345600/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.289154:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345600:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.289181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.296744:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345600:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.296835:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x2e5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.296850:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.296906:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x2e5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.296919:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x2e5:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773262.297008:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.297023:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1902, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.297028:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1902, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.297048:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1902,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.297412:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.297422:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.297432:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.297453:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345728:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.297525:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.299044:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345728:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.299060:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299148:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299156:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299170:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299180:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299198:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299205:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.299216:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664345728/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.299227:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299235:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1593 (0 0) 00000002:00010000:1.0:1775773262.299247:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x639:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.299275:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x639:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299282:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x639:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.299289:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x639:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.299303:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345856:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.299351:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.300534:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664345856:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.300547:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x639:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300557:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300566:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300576:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300586:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300593:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300599:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.300612:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3480 x1862032664345856/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.300621:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300629:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1593 (0 0) 00000080:00200000:1.0:1775773262.300644:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.300677:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x639:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.300761:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x639:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec1aa 00000080:00010000:1.0:1775773262.300766:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x639:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec1a3 00000080:00010000:1.0:1775773262.300772:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x639:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.300774:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1a3 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.300776:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300783:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.300805:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1aa from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.300807:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.300812:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.300827:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.300833:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1902, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.300867:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.300890:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.300894:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.300912:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1a3 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6e92 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.300928:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.300941:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.300944:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.300946:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1aa lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x639:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6ea0 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.300956:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.300960:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2680 x1862032664346240/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.300969:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346240:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.301008:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.310045:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346240:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.310160:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x639:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.310172:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.310220:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x639:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.310232:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x639:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.310316:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.310330:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1381, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.310334:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1381, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.310355:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1381,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.310458:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.310467:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.310476:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.310494:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346368:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.310658:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.312668:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346368:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.312687:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312706:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312712:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312723:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312731:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312738:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312743:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312747:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.312754:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0700 x1862032664346368/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.312762:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.312768:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312775:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1061 (0 0) 00000080:00200000:1.0:1775773262.312779:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.312814:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x425:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773262.312898:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x425:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.312905:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x425:0x0](ffff98297d9a7448) for lock 0x12ae90f9442ec1b1 00000080:00010000:1.0:1775773262.312912:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x425:0x0](ffff98297d9a7448) 00000080:00010000:1.0:1775773262.312915:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1b1 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.312917:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.312922:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.312940:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.312945:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1381, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.312973:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.312987:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x425:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.313011:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.313015:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.313017:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1b1 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6ef4 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.313030:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.313034:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0700 x1862032664346624/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.313054:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346624:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.313120:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.314853:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346624:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.314898:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x425:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.314908:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.314943:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x425:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.314951:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x425:0x0](ffff98297d9a7448) 00000080:00200000:1.0:1775773262.315020:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.315031:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.411, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.315034:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.411, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.315049:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.411,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.315114:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.315121:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.315133:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.315144:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346752:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.315190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.316737:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664346752:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.316749:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316769:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316778:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316788:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316810:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316820:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316827:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316834:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.316843:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2a00 x1862032664346752/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.316854:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.316864:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.316872:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/874 (0 0) 00000080:00200000:1.0:1775773262.316877:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.316902:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x36a:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.316976:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x36a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.316985:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x36a:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec1b8 00000080:00010000:1.0:1775773262.316993:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x36a:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.316997:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1b8 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.317000:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.317009:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.317027:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.317033:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.411, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.317095:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.317119:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x36a:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.317148:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.317166:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.317169:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1b8 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x36a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c6f2c expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.317187:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.317192:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2a00 x1862032664347008/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.317204:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347008:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.317242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.318741:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347008:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.318788:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x36a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.318809:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.318848:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x36a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.318858:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x36a:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.318926:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.318937:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1859, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.318941:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1859, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.318955:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1859,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.318987:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.318996:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.319004:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.319017:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347136:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.319059:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.320421:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347136:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.320431:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320446:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320452:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320459:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320465:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320470:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320475:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320480:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.320485:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2300 x1862032664347136/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.320493:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320498:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1574 (0 0) 00000002:00010000:1.0:1775773262.320507:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x626:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.320524:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x626:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320529:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x626:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.320536:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x626:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.320550:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347264:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.320587:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.322920:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347264:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.322937:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x626:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322949:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322958:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322968:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322979:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322985:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.322989:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.323000:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664347264/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.323009:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.323016:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1574 (0 0) 00000080:00200000:1.0:1775773262.323028:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.323050:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x626:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.323281:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x626:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec1c6 00000080:00010000:1.0:1775773262.323286:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x626:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec1bf 00000080:00010000:1.0:1775773262.323292:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x626:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.323295:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1bf from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.323297:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.323304:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.323312:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1c6 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.323314:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.323318:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.323332:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.323338:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1859, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.323369:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.323404:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.323408:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.323427:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1bf lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6f64 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.323442:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.323455:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.323459:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.323460:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1c6 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x626:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6f72 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.323470:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.323475:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2300 x1862032664347648/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.323485:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347648:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.323528:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.330256:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347648:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.330342:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x626:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.330354:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.330411:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x626:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.330421:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x626:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.330497:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.330513:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2372, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.330517:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2372, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.330545:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2372,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.330589:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.330598:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.330607:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.330633:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347776:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.330695:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.332112:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347776:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.332126:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332146:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332152:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332165:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332180:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332190:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332199:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332204:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.332211:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe2d80 x1862032664347776/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.332219:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332226:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1566 (0 0) 00000002:00010000:1.0:1775773262.332235:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x61e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.332254:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332266:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.332271:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.332280:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347904:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.332311:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.333208:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664347904:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.333215:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333221:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333227:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333232:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333237:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333242:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333246:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.333254:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664347904/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.333261:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333266:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1566 (0 0) 00000080:00200000:1.0:1775773262.333276:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.333295:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x61e:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.333359:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x61e:0x0](ffff98297d9a0908) for remote lock 0x12ae90f9442ec1d4 00000080:00010000:1.0:1775773262.333362:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x61e:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec1cd 00000080:00010000:1.0:1775773262.333369:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x61e:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.333372:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1cd from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.333375:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333384:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.333405:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1d4 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.333408:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.333416:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.333429:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.333434:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2372, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.333468:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.333497:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.333504:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.333524:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1cd lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6fc6 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.333551:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.333574:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.333587:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.333590:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1d4 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x61e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c6fd4 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.333607:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.333614:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe2d80 x1862032664348288/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.333640:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348288:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.333662:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.338479:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348288:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.338558:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x61e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.338568:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.338614:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x61e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.338626:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x61e:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.338709:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.338725:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.243, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.338729:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.243, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.338748:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.243,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.338804:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.338814:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.338822:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.338837:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348416:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.338886:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.339980:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348416:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.339992:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340015:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340023:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340038:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340048:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340058:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340084:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340093:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.340106:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe3480 x1862032664348416/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.340118:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340129:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/800 (0 0) 00000002:00010000:1.0:1775773262.340140:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x320:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.340165:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x320:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340172:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x320:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.340180:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x320:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.340194:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348544:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.340239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.344190:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348544:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.344209:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x320:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344221:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344231:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344241:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344252:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344270:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344278:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.344296:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664348544/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.344307:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344323:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/800 (0 0) 00000080:00200000:1.0:1775773262.344341:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.344369:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x320:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.344453:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x320:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec1e2 00000080:00010000:1.0:1775773262.344459:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x320:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec1db 00000080:00010000:1.0:1775773262.344466:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x320:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.344468:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1db from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.344472:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344481:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.344492:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1e2 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.344495:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.344501:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.344520:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.344528:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.243, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.344565:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.344605:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.344612:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.344634:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1db lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7028 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.344665:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.344689:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.344696:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.344698:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1e2 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x320:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7036 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.344716:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.344723:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe3480 x1862032664348928/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.344739:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348928:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.344769:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.348855:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664348928:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.348927:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x320:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.348940:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.349006:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x320:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.349020:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x320:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.349131:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.349146:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1845, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.349159:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1845, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.349181:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1845,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.349235:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.349245:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.349256:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.349272:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349056:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.349340:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.350821:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349056:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.350835:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350857:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350865:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350880:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350889:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350899:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350906:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350923:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.350933:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982981fe0380 x1862032664349056/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.350945:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.350955:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1567 (0 0) 00000002:00010000:1.0:1775773262.350971:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x61f:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.350997:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.351004:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.351011:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.351024:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349184:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.351101:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.352142:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349184:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.352165:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352174:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352181:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352190:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352199:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352206:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352214:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.352239:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9829745c7800 x1862032664349184/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.352249:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352257:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1567 (0 0) 00000080:00200000:1.0:1775773262.352272:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.352298:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x61f:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.352385:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x61f:0x0](ffff98297d9a0908) for remote lock 0x12ae90f9442ec1f0 00000080:00010000:1.0:1775773262.352391:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x61f:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec1e9 00000080:00010000:1.0:1775773262.352398:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x61f:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.352401:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1e9 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.352404:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352413:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.352423:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1f0 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.352426:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.352432:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.352450:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.352464:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1845, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.352507:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.352557:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.352565:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.352590:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec1e9 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c708a expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.352611:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.352634:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.352640:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.352643:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec1f0 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x61f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7098 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.352668:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.352675:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982981fe0380 x1862032664349568/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.352689:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349568:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.352739:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.357616:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349568:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.357678:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x61f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.357699:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.357750:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x61f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.357764:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x61f:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.357862:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.357874:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1890, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.357877:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1890, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.357894:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1890,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.357932:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.357946:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.357953:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.357966:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349696:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.358013:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.359900:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349696:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.359912:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359932:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359940:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359955:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359965:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359976:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359983:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.359988:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.359998:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4f100 x1862032664349696/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.360009:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.360018:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1326 (0 0) 00000002:00010000:1.0:1775773262.360031:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x52e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.360057:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.360092:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.360102:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.360117:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4dc00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349824:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.360160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.361169:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4dc00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664349824:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.361180:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361189:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361197:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361205:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361212:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361216:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361220:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.361239:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4dc00 x1862032664349824/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.361249:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361257:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1326 (0 0) 00000080:00200000:1.0:1775773262.361271:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.361297:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x52e:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.361386:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x52e:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec1fe 00000080:00010000:1.0:1775773262.361399:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x52e:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec1f7 00000080:00010000:1.0:1775773262.361406:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x52e:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.361408:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec1f7 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.361411:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361419:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.361429:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec1fe from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.361432:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.361439:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.361462:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.361469:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1890, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.361508:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.361546:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.361554:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.361575:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec1f7 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c70ec expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.361596:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.361625:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.361639:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.361641:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec1fe lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x52e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c70fa expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.361658:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.361666:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4f100 x1862032664350208/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.361681:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350208:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.361710:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.366764:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350208:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.366846:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x52e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.366856:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.366904:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x52e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.366917:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x52e:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.367006:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.367019:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1297, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.367022:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1297, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.367041:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1297,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.367112:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.367119:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.367126:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.367140:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350336:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.367198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.368926:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350336:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.368943:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.368967:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.368976:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.368993:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369003:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369016:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369025:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369032:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.369055:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4d500 x1862032664350336/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.369086:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369094:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1306 (0 0) 00000002:00010000:1.0:1775773262.369105:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x51a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.369129:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369134:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.369140:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.369165:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350464:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.369217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.391650:0:412873:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862017593095168 00000100:00100000:2.0:1775773262.391659:0:412873:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862017593095168 00000100:00100000:2.0:1775773262.391712:0:412873:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.118@tcp, seq: 296 00000100:00100000:2.0:1775773262.391722:0:412873:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff982981c35f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:220608:x1862017593095168:12345-192.168.201.118@tcp:104: 00010000:00010000:2.0:1775773262.391740:0:412873:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff982942b6f800 lock: 00000000fd62ede5/0x12ae90f9442ebc94 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0xb3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x96e4bd9bff2c4bd8 expref: -99 pid: 412927 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773262.391768:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ca80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350464:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.391786:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391811:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391822:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391833:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391844:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391852:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391859:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.391875:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4ca80 x1862032664350464/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.391888:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.391902:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1306 (0 0) 00010000:00010000:2.0:1775773262.391906:0:412873:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1775773262.391914:0:412873:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff982981c35f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:220608:x1862017593095168:12345-192.168.201.118@tcp:104: Request processed in 192us (326us total) trans 0 rc 0/0 00000080:00200000:1.0:1775773262.391921:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000100:00100000:2.0:1775773262.391926:0:412873:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.118@tcp, seq: 296 00000080:00200000:1.0:1775773262.391954:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x51a:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.392047:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x51a:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec20c 00000080:00010000:1.0:1775773262.392056:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x51a:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec205 00000080:00010000:1.0:1775773262.392095:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x51a:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.392099:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec205 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.392103:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.392113:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.392128:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec20c from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.392132:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.392139:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.392162:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.392171:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1297, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.392229:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.392263:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.392270:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.392277:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec205 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c714e expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.392299:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.392326:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.392332:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.392334:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec20c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x51a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c715c expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.392352:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.392359:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4d500 x1862032664350976/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.392375:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350976:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.392402:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.412514:0:412860:0:(client.c:3082:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff982942b6f800: committing for last_committed 12884953615 gen 1 00000100:00100000:1.0:1775773262.413027:0:412860:0:(client.c:3126:ptlrpc_free_committed()) @@@ stopping search req@ffff982977d4d500 x1862032664350976/t12884953617(12884953617) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/456 e 0 to 0 dl 1775773303 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773262.413041:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664350976:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.413173:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x51a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.413185:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.413245:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x51a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.413258:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x51a:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.413339:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.413357:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1430, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.413362:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1430, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.413382:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1430,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.413429:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.413438:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.413448:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.413466:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351104:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.413528:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.415504:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351104:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.415520:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415539:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415549:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415556:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415563:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415571:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415578:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415584:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.415591:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4df80 x1862032664351104/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.415597:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.415605:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415615:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1092 (0 0) 00000080:00200000:1.0:1775773262.415619:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.415642:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x444:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.415724:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x444:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.415731:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x444:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec213 00000080:00010000:1.0:1775773262.415737:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x444:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.415739:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec213 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.415742:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.415748:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.415764:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.415769:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1430, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.415812:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.415838:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x444:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.415875:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.415882:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.415885:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec213 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7377 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.415905:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.415912:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4df80 x1862032664351360/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.415931:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351360:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.415989:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.419127:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4df80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351360:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.419193:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x444:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.419205:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.419262:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x444:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.419275:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x444:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.419354:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.419368:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1560, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.419372:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1560, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.419392:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1560,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.419436:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.419445:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.419454:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.419472:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351488:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.419531:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.422481:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351488:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.422503:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422531:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422540:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422551:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422562:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422574:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422582:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422589:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.422599:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c700 x1862032664351488/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.422612:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422621:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1162 (0 0) 00000002:00010000:1.0:1775773262.422637:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x48a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.422675:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422682:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.422691:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.422709:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351616:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.422772:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.424649:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664351616:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.424667:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424680:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424689:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424700:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424711:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424719:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424727:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.424737:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4fb80 x1862032664351616/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.424749:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424759:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1162 (0 0) 00000080:00200000:1.0:1775773262.424776:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.424825:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x48a:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.424922:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x48a:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec221 00000080:00010000:1.0:1775773262.424928:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x48a:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec21a 00000080:00010000:1.0:1775773262.424936:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x48a:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.424940:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec21a from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.424943:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424953:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.424965:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec221 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.424968:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.424974:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.425001:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.425009:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1560, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.425054:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.425116:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.425124:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.425144:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec21a lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c73af expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.425169:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.425195:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.425203:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.425205:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec221 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x48a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c73bd expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.425223:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.425230:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4c700 x1862032664352000/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.425247:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352000:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.426454:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.428512:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664351744:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.442855:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352000:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.442943:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x48a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.442954:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.443008:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x48a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.443017:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x48a:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.443129:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.443141:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2281, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.443153:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2281, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.443177:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2281,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.443232:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.443242:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.443251:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.443267:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352128:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.443324:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.444842:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352128:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.444854:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444875:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444880:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444888:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444895:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444901:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444906:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444911:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.444920:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4fb80 x1862032664352128/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.444930:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.444945:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.444955:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1786 (0 0) 00000080:00200000:1.0:1775773262.444961:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.444985:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x6fa:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.445111:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x6fa:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.445125:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x6fa:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec228 00000080:00010000:1.0:1775773262.445135:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x6fa:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.445138:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec228 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.445141:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.445150:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.445177:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.445184:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2281, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.445225:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.445246:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x6fa:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.445281:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.445288:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.445290:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec228 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x6fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7411 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.445309:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.445316:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4fb80 x1862032664352384/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.445333:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352384:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.445391:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.448743:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4fb80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352384:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.448838:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x6fa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.448851:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.448901:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x6fa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.448915:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x6fa:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.449000:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.449016:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1446, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.449021:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1446, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.449042:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1446,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.449119:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.449130:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.449140:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.449175:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ed80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352512:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.449243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.456991:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ed80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352512:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.457012:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457041:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457049:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457057:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457087:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457098:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457103:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457107:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.457115:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4ed80 x1862032664352512/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.457125:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457133:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1368 (0 0) 00000002:00010000:1.0:1775773262.457153:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x558:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.457185:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x558:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457190:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x558:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.457196:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x558:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.457209:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352640:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.457273:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.459432:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664352640:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.459450:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x558:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459461:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459472:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459482:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459495:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459503:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459510:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.459520:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c380 x1862032664352640/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.459533:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459543:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1368 (0 0) 00000080:00200000:1.0:1775773262.459561:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.459594:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x558:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.459704:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x558:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec236 00000080:00010000:1.0:1775773262.459710:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x558:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec22f 00000080:00010000:1.0:1775773262.459717:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x558:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.459720:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec22f from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.459724:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459738:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.459751:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec236 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.459754:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.459761:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.459780:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.459807:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1446, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.459856:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.459896:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.459902:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.459930:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7a00/0x12ae90f9442ec22f lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7449 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.459978:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.460007:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.460014:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.460017:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec236 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x558:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7457 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.460036:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.460045:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4ed80 x1862032664353024/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.460062:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ed80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353024:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.460118:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.469390:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ed80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353024:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.469487:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x558:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.469499:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.469564:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x558:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.469579:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x558:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.469670:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.469699:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1823, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.469704:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1823, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.469727:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1823,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.469783:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.469806:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.469816:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.469836:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353152:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.469902:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.473586:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353152:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.473605:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473629:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473634:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473642:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473649:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473659:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473665:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473669:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.473675:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4f480 x1862032664353152/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.473683:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473690:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1287 (0 0) 00000002:00010000:1.0:1775773262.473781:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x507:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.473839:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x507:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473847:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x507:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.473855:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x507:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.473873:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353280:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.473932:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.476389:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353280:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.476409:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x507:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476420:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476430:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476441:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476453:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476460:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476478:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.476489:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4e680 x1862032664353280/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.476502:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476513:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1287 (0 0) 00000080:00200000:1.0:1775773262.476531:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.476562:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x507:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.476654:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x507:0x0](ffff98297d9a0908) for remote lock 0x12ae90f9442ec244 00000080:00010000:1.0:1775773262.476659:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x507:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec23d 00000080:00010000:1.0:1775773262.476667:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x507:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.476670:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec23d from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.476674:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476823:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.476838:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec244 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.476841:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.476848:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.476868:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.476876:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1823, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.476923:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.476960:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.476966:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.477213:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec23d lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c74ab expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.477238:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.477261:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.477266:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.477269:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec244 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x507:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c74b9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.477296:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.477302:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4f480 x1862032664353664/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.477318:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353664:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.477343:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.484443:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f480 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353664:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.484523:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x507:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.484535:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.484586:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x507:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.484599:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x507:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.484681:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.484707:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2234, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.484711:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2234, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.484732:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2234,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.484781:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.484801:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.484811:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.484830:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353792:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.484900:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.489760:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353792:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.489799:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489836:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489845:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489857:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489868:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489882:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489890:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489898:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.489910:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4d180 x1862032664353792/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.489922:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.489932:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1493 (0 0) 00000002:00010000:1.0:1775773262.489948:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x5d5:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.489993:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.490001:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.490010:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.490032:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353920:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.490371:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.494021:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664353920:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.494043:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494400:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494412:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494422:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494435:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494443:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494451:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.494462:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c000 x1862032664353920/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.494642:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494654:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1493 (0 0) 00000080:00200000:1.0:1775773262.494672:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.494859:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x5d5:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.494963:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x5d5:0x0](ffff98297d9a0908) for remote lock 0x12ae90f9442ec252 00000080:00010000:1.0:1775773262.494969:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x5d5:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec24b 00000080:00010000:1.0:1775773262.494977:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x5d5:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.494980:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec24b from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.494984:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.494993:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.495005:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec252 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.495009:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.495016:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.495034:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.495042:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2234, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.495403:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.495443:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.495450:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.496176:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec24b lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c750d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.496206:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.496248:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.496254:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.496257:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec252 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x5d5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c751b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.496283:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.496292:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4d180 x1862032664354304/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.496310:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4d180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354304:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.496396:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.511775:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4d180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354304:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.511882:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x5d5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.511893:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.511945:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x5d5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.511960:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x5d5:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.512061:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.513393:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.971, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.513400:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.971, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.513426:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.971,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.513500:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.513512:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.513522:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.513552:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354432:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.513628:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.516930:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354432:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.516955:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.516984:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.516993:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517004:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517019:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517030:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517206:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517214:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.517225:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94e00 x1862032664354432/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.517235:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773262.517246:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517255:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1127 (0 0) 00000080:00200000:2.0:1775773262.517261:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.517293:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x467:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.517430:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x467:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773262.517440:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x467:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec259 00000080:00010000:2.0:1775773262.517450:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x467:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.517461:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec259 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.517464:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.517470:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.517493:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.517499:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.971, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.517535:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.517556:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x467:0x0]: losing layout lock 00010000:00010000:2.0:1775773262.517595:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.517602:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.517605:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec259 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x467:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c756f expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.517626:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.517633:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f94e00 x1862032664354688/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.517654:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354688:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.517713:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.521253:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94e00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354688:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.521332:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x467:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.521345:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.521412:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x467:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.521424:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x467:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.521523:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.521540:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1784, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.521544:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1784, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.521565:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1784,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.521615:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.521625:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.521634:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.521652:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354816:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.521716:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.526941:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354816:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.526965:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.526994:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527002:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527014:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527021:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527030:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527044:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527048:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.527056:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f97100 x1862032664354816/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.527184:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527195:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1268 (0 0) 00000002:00010000:2.0:1775773262.527208:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x4f4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.527241:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527246:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.527252:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.527267:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354944:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.527321:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.529509:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664354944:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.529527:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529548:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529559:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529569:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529581:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529589:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529596:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.529612:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f96d80 x1862032664354944/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.529624:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529636:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1268 (0 0) 00000080:00200000:2.0:1775773262.529654:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.529697:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x4f4:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773262.529809:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4f4:0x0](ffff98296faec2c8) for remote lock 0x12ae90f9442ec267 00000080:00010000:2.0:1775773262.529814:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x4f4:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ec260 00000080:00010000:2.0:1775773262.529822:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x4f4:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773262.529825:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec260 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.529829:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529838:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.529850:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec267 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.529853:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.529859:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.529876:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.529884:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1784, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.529927:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.529961:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.529967:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.529994:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec260 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c75a7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775773262.530018:0:404332:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1775773262.530031:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664355072:192.168.201.118@tcp:103:rm.0 00010000:00010000:2.0:1775773262.530988:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.531018:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.531025:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.531027:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec267 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x4f4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c75b5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.531047:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.531094:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f97100 x1862032664355328/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.531111:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355328:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.531138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.533866:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664355072:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.538737:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355328:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.538837:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x4f4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.538848:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.538898:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x4f4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.538910:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x4f4:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773262.538995:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.539011:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1047, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.539016:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1047, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.539038:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1047,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.540400:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.540413:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.540423:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.540447:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355456:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.540530:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.542306:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355456:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.542324:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542348:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542362:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542375:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542386:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542398:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542406:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542414:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.542425:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f96d80 x1862032664355456/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.542437:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542447:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1164 (0 0) 00000002:00010000:2.0:1775773262.542461:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x48c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.542496:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542504:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.542512:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.542525:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355584:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.542573:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.543734:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355584:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.543743:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543752:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543761:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543770:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543779:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543805:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543818:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.543825:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94a80 x1862032664355584/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.543833:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543839:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1164 (0 0) 00000080:00200000:2.0:1775773262.543862:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.543889:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x48c:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.543960:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x48c:0x0](ffff98296faeba88) for remote lock 0x12ae90f9442ec275 00000080:00010000:2.0:1775773262.543964:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x48c:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec26e 00000080:00010000:2.0:1775773262.543969:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x48c:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.543971:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec26e from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.543973:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543980:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.543987:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec275 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.543989:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.543993:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.544006:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.544012:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1047, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.544044:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.544466:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.544472:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.544489:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec26e lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7609 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.544507:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.544524:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.544535:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.544540:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec275 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x48c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7617 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.544553:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.544558:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f96d80 x1862032664355968/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.544569:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355968:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.544596:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.549021:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664355712:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.553369:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96d80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664355968:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.553451:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x48c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.553583:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.553637:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x48c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.553650:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x48c:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.553933:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.553948:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1188, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.553952:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1188, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.553974:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1188,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.554025:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.554036:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.554044:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.554089:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356096:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.554160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.556313:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356096:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.556330:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556357:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556365:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556375:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556386:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556397:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556405:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556412:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.556422:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94a80 x1862032664356096/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.556432:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773262.556449:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556459:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1245 (0 0) 00000080:00200000:2.0:1775773262.556475:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.556502:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x4dd:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773262.556613:0:412860:0:(file.c:6766:ll_layout_conf()) [0x2000013a1:0x4dd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773262.556624:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4dd:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ec27c 00000080:00010000:2.0:1775773262.556633:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4dd:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773262.556637:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec27c from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.556640:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.556649:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.556674:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.556690:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1188, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.556732:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.556754:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x4dd:0x0]: losing layout lock 00010000:00010000:2.0:1775773262.556802:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.556809:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.556812:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec27c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4dd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c766b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.556835:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.556841:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f94a80 x1862032664356352/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.556859:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356352:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.556915:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.563972:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94a80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356352:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.564053:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x4dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.564080:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.564130:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x4dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.564143:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x4dd:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773262.564230:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.564245:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2251, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.564250:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2251, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.564270:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2251,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.564320:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.564330:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.564341:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.564360:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356480:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.564427:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.566399:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356480:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.566419:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566442:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566450:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566461:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566471:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566481:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566488:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566493:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.566502:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f94700 x1862032664356480/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.566512:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775773262.566524:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566533:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1503 (0 0) 00000080:00200000:2.0:1775773262.566539:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.566568:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x5df:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.566665:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x5df:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775773262.566672:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x5df:0x0](ffff98296faeba88) for lock 0x12ae90f9442ec283 00000080:00010000:2.0:1775773262.566680:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x5df:0x0](ffff98296faeba88) 00000080:00010000:2.0:1775773262.566682:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec283 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.566685:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.566691:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.566711:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.566716:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2251, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.566746:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.566760:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x5df:0x0]: losing layout lock 00010000:00010000:2.0:1775773262.566785:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.566812:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.566815:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec283 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x5df:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c76a3 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.566836:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.566842:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f94700 x1862032664356736/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.566862:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356736:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.566955:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.569355:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94700 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356736:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.569424:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x5df:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.569435:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.569489:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x5df:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.569501:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x5df:0x0](ffff98296faeba88) 00000080:00200000:2.0:1775773262.569598:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.569614:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1492, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.569618:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1492, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.569636:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1492,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.569685:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.569695:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.569704:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.569739:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356864:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.569803:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.571873:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356864:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.571889:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571908:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571914:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571922:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571932:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571952:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571960:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.571974:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.571991:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f95500 x1862032664356864/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.572002:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.572012:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1388 (0 0) 00000002:00010000:2.0:1775773262.572025:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x56c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.572058:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.572092:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.572101:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.572116:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356992:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.572169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.573379:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664356992:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.573391:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573400:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573406:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573413:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573420:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573424:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573428:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.573437:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f95180 x1862032664356992/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.573448:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573457:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1388 (0 0) 00000080:00200000:2.0:1775773262.573482:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775773262.573511:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x56c:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773262.573612:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x56c:0x0](ffff98296faec2c8) for remote lock 0x12ae90f9442ec291 00000080:00010000:2.0:1775773262.573618:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x56c:0x0](ffff98296faec2c8) for lock 0x12ae90f9442ec28a 00000080:00010000:2.0:1775773262.573626:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x56c:0x0](ffff98296faec2c8) 00000080:00010000:2.0:1775773262.573629:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec28a from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.573633:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573642:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1775773262.573652:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec291 from it ffffa414081d3df8 00010000:00010000:2.0:1775773262.573655:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.573662:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775773262.573677:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:2.0:1775773262.573683:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1492, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:2.0:1775773262.573719:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.573750:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.573757:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.573784:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6a00/0x12ae90f9442ec28a lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c76db expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.573835:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.573856:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775773262.573863:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775773262.573865:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7a00/0x12ae90f9442ec291 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x56c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c76e9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1775773262.573882:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1775773262.573889:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982971f95500 x1862032664357376/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773262.573903:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357376:192.168.201.118@tcp:36:rm.0 00000100:00100000:2.0:1775773262.573933:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d9c0 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.575499:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664357120:192.168.201.118@tcp:103:rm.0 00000100:00100000:2.0:1775773262.581102:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95500 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357376:192.168.201.118@tcp:36:rm.0 00000080:00200000:2.0:1775773262.581186:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x56c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1775773262.581198:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775773262.581247:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x56c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775773262.581258:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x56c:0x0](ffff98296faec2c8) 00000080:00200000:2.0:1775773262.581351:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:2.0:1775773262.581368:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.611, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:2.0:1775773262.581372:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.611, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:2.0:1775773262.581395:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.611,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.581446:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.581456:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.581486:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.581506:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357504:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.581583:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.585441:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357504:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.585473:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585501:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585509:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585519:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585529:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585541:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585548:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585561:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.585571:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f95180 x1862032664357504/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.585581:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff982965dc7c00/0x12ae90f9442ec298 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c773d expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585590:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/657 (0 0) 00000002:00010000:2.0:1775773262.585603:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x291:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775773262.585642:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x291:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585649:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x291:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.585656:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x291:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775773262.585674:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357632:192.168.201.118@tcp:101:rm.0 00000100:00100000:2.0:1775773262.585742:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c15d480 going to sleep for 36 seconds 00000100:00100000:2.0:1775773262.590102:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95f80 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664357632:192.168.201.118@tcp:101:rm.0 00010000:00010000:2.0:1775773262.590121:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x291:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590130:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590139:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590147:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590158:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590166:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590173:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775773262.590184:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982971f95f80 x1862032664357632/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775773262.590197:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff982965dc6800/0x12ae90f9442ec29f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x291:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c774b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775773262.590210:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/657 (0 0) 00000080:00200000:2.0:1775773262.590227:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000100:00100000:1.0:1775773262.603770:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664358784:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.605048:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664358784:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.616400:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664359808:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.617491:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664359808:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.625092:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664360448:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.626577:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982965ec3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664360448:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.649754:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664361856:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.652623:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664361856:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.673492:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664362880:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.673964:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db10c0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.675129:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664362880:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.682813:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7100 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363136:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.682899:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x51c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.682911:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.682961:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x51c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.682974:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x51c:0x0](ffff982952593248) 00000080:00200000:1.0:1775773262.683096:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.683112:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.683116:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.683139:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.874,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.683187:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.683197:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.683206:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.683226:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363264:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.683291:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.685993:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363264:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.686012:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686040:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686049:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686060:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686086:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686098:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686107:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686113:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.686125:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4ce00 x1862032664363264/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.686135:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.686146:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686155:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/808 (0 0) 00000080:00200000:1.0:1775773262.686161:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.686198:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x328:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.686318:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x328:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.686329:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x328:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec30f 00000080:00010000:1.0:1775773262.686339:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x328:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.686342:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec30f from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.686346:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.686354:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.686382:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.686389:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.874, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.686430:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.686452:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x328:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.686487:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.686494:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.686497:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec30f lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x328:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7aa1 expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.686516:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.686523:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4ce00 x1862032664363520/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.686544:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363520:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.686621:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.689773:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ce00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363520:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.689865:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x328:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.689876:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.689924:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x328:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.689938:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x328:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.690021:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.690036:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.2040, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.690041:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.2040, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.690062:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.2040,[0x0:0x0:0x0]) in obj [0x2000013a2:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.690143:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.690153:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.690163:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 3/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.690190:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ea00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363648:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.690254:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.693139:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ea00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363648:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.693160:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: --/CR res: [0x2000013a2:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693198:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693208:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693219:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693230:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693243:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693250:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693257:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.693268:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4ea00 x1862032664363648/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.693278:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693288:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 9663679440/1397 (0 0) 00000002:00010000:1.0:1775773262.693312:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000bd0:0x575:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.693350:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x575:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693358:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x575:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.693367:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 3/1,0 mode: --/CR res: [0x240000bd0:0x575:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.693384:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363776:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.693446:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.695819:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664363776:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.695838:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 4/1,0 mode: --/CR res: [0x240000bd0:0x575:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695847:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695854:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695861:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695870:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695875:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695879:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.695889:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4f800 x1862032664363776/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.695898:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.695906:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/1397 (0 0) 00000080:00200000:1.0:1775773262.695920:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x2000013a2:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.695949:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x575:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.696027:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x575:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec31d 00000080:00010000:1.0:1775773262.696031:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x575:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec316 00000080:00010000:1.0:1775773262.696036:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x575:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.696039:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec316 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.696041:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.696048:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.696056:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec31d from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.696058:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.696063:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.696103:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.696110:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.2040, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.696146:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.696172:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.696185:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.696204:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec316 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7ad9 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775773262.696230:0:404330:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775773262.696243:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664363904:192.168.201.118@tcp:103:rm.0 00010000:00010000:1.0:1775773262.696311:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.696337:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.696343:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.696345:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec31d lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x575:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7ae7 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.696363:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.696369:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4ea00 x1862032664364160/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.696384:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4ea00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364160:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.696406:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.699134:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664363904:192.168.201.118@tcp:103:rm.0 00000100:00100000:1.0:1775773262.710735:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4ea00 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364160:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.710819:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x575:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.710830:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.710880:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x575:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.710893:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x575:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.710980:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.710994:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.156, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.710998:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.156, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.711019:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.156,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.711109:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.711119:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.711130:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.711149:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364288:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.711209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.714674:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364288:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.714707:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714736:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714749:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: --/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714761:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714772:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714794:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714805:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 5/2,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714812:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 4/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.714823:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4f800 x1862032664364288/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.714835:0:412860:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775773262.714849:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.714864:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 9663679440/417 (0 0) 00000080:00200000:1.0:1775773262.714870:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.714903:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240000bd0:0x1a1:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.715018:0:412860:0:(file.c:6766:ll_layout_conf()) [0x240000bd0:0x1a1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775773262.715029:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240000bd0:0x1a1:0x0](ffff98297d9a0908) for lock 0x12ae90f9442ec324 00000080:00010000:1.0:1775773262.715039:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000bd0:0x1a1:0x0](ffff98297d9a0908) 00000080:00010000:1.0:1775773262.715042:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec324 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.715045:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 3/1,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.715052:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.715111:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.715120:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.156, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.715170:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 2/0,0 mode: PR/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.715190:0:412860:0:(vvp_object.c:117:vvp_conf_set()) [0x240000bd0:0x1a1:0x0]: losing layout lock 00010000:00010000:1.0:1775773262.715225:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 1/0,0 mode: --/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.715229:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.715231:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec324 lrc: 0/0,0 mode: --/PR res: [0x240000bd0:0x1a1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff2c7b3b expref: -99 pid: 412860 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.715245:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.715250:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4f800 x1862032664364544/t0(0) o36->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.715270:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364544:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.715338:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.718622:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4f800 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364544:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.718705:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x240000bd0:0x1a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.718717:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.718766:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x240000bd0:0x1a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.718779:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240000bd0:0x1a1:0x0](ffff98297d9a0908) 00000080:00200000:1.0:1775773262.718898:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.718914:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1593, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.718919:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1593, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.718942:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1593,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.718995:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.719006:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.719016:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.719035:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4e300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364672:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.719134:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.720428:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4e300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364672:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.720443:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720476:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720483:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720493:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720503:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720514:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720522:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720529:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.720549:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4e300 x1862032664364672/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.720560:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720570:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1441 (0 0) 00000002:00010000:1.0:1775773262.720582:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x5a1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.720611:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720619:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.720626:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.720639:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364800:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.720685:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.722396:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c000 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664364800:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.722413:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722424:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722433:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722443:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722452:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722459:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722466:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.722476:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c000 x1862032664364800/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.722486:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722495:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1441 (0 0) 00000080:00200000:1.0:1775773262.722587:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.722618:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x5a1:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.722728:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5a1:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec332 00000080:00010000:1.0:1775773262.722735:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5a1:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec32b 00000080:00010000:1.0:1775773262.722743:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5a1:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.722746:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec32b from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.722750:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722760:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.722773:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec332 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.722777:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.722794:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.722813:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.722821:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1593, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.722864:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.722896:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.722903:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.723186:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f7000/0x12ae90f9442ec32b lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7b73 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.723210:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.723235:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.723241:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.723243:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec332 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5a1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7b81 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.723338:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.723345:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4e300 x1862032664365184/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.723360:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4e300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365184:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.723395:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.731507:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4e300 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365184:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.731598:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x5a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.731610:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.731730:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x5a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.731743:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x5a1:0x0](ffff98297d9a5b88) 00000080:00200000:1.0:1775773262.731846:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775773262.731862:0:412860:0:(namei.c:1356:ll_lookup_nd()) VFS Op:name=f81b.sanityn-new.1814, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), flags=0 00000080:00200000:1.0:1775773262.731867:0:412860:0:(namei.c:1097:ll_lookup_it()) VFS Op:name=f81b.sanityn-new.1814, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8), intent=getattr 00000002:00010000:1.0:1775773262.731889:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f81b.sanityn-new.1814,[0x0:0x0:0x0]) in obj [0x240000bd1:0x1e:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.731937:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.731947:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.731957:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 3/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.731975:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365312:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.732043:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.737399:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365312:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.737426:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: --/CR res: [0x240000bd1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737457:0:412860:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737471:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737481:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737492:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737504:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737513:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737521:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.737532:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4e680 x1862032664365312/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/584 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.737544:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737554:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829730f9400 type 13 mode 4 res 8589939617/1555 (0 0) 00000002:00010000:1.0:1775773262.737579:0:412860:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x613:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775773262.737623:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x613:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737632:0:412860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x613:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.737641:0:412860:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x613:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 412860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775773262.737671:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4c380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365440:192.168.201.118@tcp:101:rm.0 00000100:00100000:1.0:1775773262.737748:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54a480 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.741465:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c380 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365440:192.168.201.118@tcp:101:rm.0 00010000:00010000:1.0:1775773262.741485:0:412860:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x613:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741498:0:412860:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741509:0:412860:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741519:0:412860:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741531:0:412860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741539:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741552:0:412860:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775773262.741563:0:412860:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff982977d4c380 x1862032664365440/t0(0) o101->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 576/640 e 0 to 0 dl 1775773303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775773262.741588:0:412860:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741600:0:412860:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9829732e4400 type 13 mode 4 res 8589939617/1555 (0 0) 00000080:00200000:1.0:1775773262.741617:0:412860:0:(namei.c:977:ll_intent_lock()) intent lock 8 on i1 [0x240000bd1:0x1e:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775773262.741657:0:412860:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x613:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.741764:0:412860:0:(llite_internal.h:1976:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x613:0x0](ffff98297d9a5b88) for remote lock 0x12ae90f9442ec340 00000080:00010000:1.0:1775773262.741770:0:412860:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x613:0x0](ffff98297d9a5b88) for lock 0x12ae90f9442ec339 00000080:00010000:1.0:1775773262.741778:0:412860:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x613:0x0](ffff98297d9a5b88) 00000080:00010000:1.0:1775773262.741781:0:412860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x12ae90f9442ec339 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.741797:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741807:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1775773262.741820:0:412860:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x12ae90f9442ec340 from it ffffa414081d3df8 00010000:00010000:1.0:1775773262.741823:0:412860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775773262.741830:0:412860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773262.741850:0:412860:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240000bd0:0x152:0x0](ffff9829525921c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775773262.741857:0:412860:0:(namei.c:2429:ll_unlink()) VFS Op:name=f81b.sanityn-new.1814, dir=[0x240000bd0:0x152:0x0](ffff9829525921c8) 00010000:00010000:1.0:1775773262.741903:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.741940:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.741947:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.741979:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff982942b6f800 lock: ffff9829466f6800/0x12ae90f9442ec339 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7bd5 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.742002:0:412860:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.742024:0:412860:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773262.742030:0:412860:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773262.742033:0:412860:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff982942b6f800 lock: ffff9829466f6a00/0x12ae90f9442ec340 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x613:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff2c7be3 expref: -99 pid: 412860 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775773262.742050:0:412860:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff982942b6f800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775773262.742057:0:412860:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff982977d4e680 x1862032664365824/t0(0) o36->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 504/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773262.742113:0:412860:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365824:192.168.201.118@tcp:36:rm.0 00000100:00100000:1.0:1775773262.742183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff98297c54acc0 going to sleep for 36 seconds 00000100:00100000:1.0:1775773262.752904:0:412860:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4e680 pname:cluuid:pid:xid:nid:opc:job rm:14dd209b-d605-4024-a2b7-27ec435c0914:412860:1862032664365824:192.168.201.118@tcp:36:rm.0 00000080:00200000:1.0:1775773262.752994:0:412860:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a1:0x613:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775773262.753014:0:412860:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775773262.753124:0:412860:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a1:0x613:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775773262.753141:0:412860:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x613:0x0](ffff98297d9a5b88) 00000100:00100000:1.0:1775773265.303281:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2585f5f-e151-4a1a-b002-2b900558e303:404330:1862032664366080:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773265.303368:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664366336:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773265.303435:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664366592:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773265.303478:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:df0ee634-793c-4832-831b-03749b568504:404330:1862032664366848:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773270.423283:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c34a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2585f5f-e151-4a1a-b002-2b900558e303:404332:1862032664366976:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773270.423377:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c34e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664367232:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773270.423420:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c36300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664367488:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773275.543367:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2585f5f-e151-4a1a-b002-2b900558e303:404329:1862032664367872:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773275.543464:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982965ec2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664368128:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773275.543503:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664368384:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773275.543547:0:404329:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:df0ee634-793c-4832-831b-03749b568504:404329:1862032664368640:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773280.663356:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664368896:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773280.663443:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664369152:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773280.663517:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664369408:192.168.201.118@tcp:400:kworker.0 00000100:00000400:1.0:1775773281.687227:0:404329:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773265/real 1775773265] req@ffff9829745c4000 x1862032664366464/t0(0) o400->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773281 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773281.687288:0:404329:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9829745c4000 x1862032664366464/t0(0) o400->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773281 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773281.687297:0:404329:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1775773281.687301:0:404329:0:(import.c:181:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff982942b6f800: Connection to lustre-OST0000 (at 192.168.201.118@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1775773281.687335:0:404329:0:(import.c:416:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1775773281.687468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773281.687474:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773281.687478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773281.687486:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 14797 00000100:00080000:2.0:1775773281.687504:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773281.687517:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773281.687540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982971f94000 x1862032664370048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773281.687587:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664370048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:2.0:1775773281.687626:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664369792:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773281.687682:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982971f95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664370176:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773281.687725:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:df0ee634-793c-4832-831b-03749b568504:404331:1862032664370432:192.168.201.118@tcp:400:kworker.0 00000100:00080000:2.0:1775773281.706983:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773281.706991:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773281.706995:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773281.707009:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 14512 00000100:00100000:1.0:1775773281.707019:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2585f5f-e151-4a1a-b002-2b900558e303:404329:1862032664367872:192.168.201.118@tcp:400:kworker.0 00000100:00080000:2.0:1775773281.707024:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773281.707078:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2585f5f-e151-4a1a-b002-2b900558e303:404331:1862032664368768:192.168.201.118@tcp:400:kworker.0 00000008:00100000:2.0:1775773281.707120:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773281.707155:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664370560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773281.707183:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773281.707186:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773281.707187:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773281.707190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 14511 00000100:00080000:2.0:1775773281.707194:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773281.707204:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664370688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773281.707216:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664370560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773281.707314:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664370688:192.168.201.118@tcp:250:kworker.0 00000100:00000400:2.0:1775773286.807184:0:404331:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773270/real 1775773270] req@ffff982981c37800 x1862032664367360/t0(0) o400->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773286.807229:0:404331:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c37800 x1862032664367360/t0(0) o400->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773286 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773286.807241:0:404331:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff982945b5e000 already not connected (conn 8, was 7): CONNECTING 00000100:00080000:1.0:1775773286.807315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773286.807320:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773286.807323:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773286.807327:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1775773286.807361:0:404330:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982977d4dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:df0ee634-793c-4832-831b-03749b568504:404330:1862032664371072:192.168.201.118@tcp:400:kworker.0 00000100:00000400:1.0:1775773291.927190:0:404329:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773275/real 1775773275] req@ffff982974b2d500 x1862032664368384/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773291.927229:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff9829764b5500 x1862032664370688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773291.927251:0:404329:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982974b2d500 x1862032664368384/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773291.927258:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664370688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773291.927293:0:404329:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff98294542a800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1775773291.927300:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff9829764b6680 x1862032664370560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773291.927311:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664368384:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773291.927316:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664370560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773291.927335:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff982971f94000 x1862032664370048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773291.927353:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982971f94000 x1862032664370048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773291 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773291.927370:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773291.927393:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773291.927399:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664370688:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773291.927407:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773291.927416:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773291.927420:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664370560:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775773291.927444:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773291.927461:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773291.927469:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664370048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773291.927484:0:404329:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:14dd209b-d605-4024-a2b7-27ec435c0914:404329:1862032664366464:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773291.927506:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664371840:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773291.927533:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664368256:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773291.927571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664371968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:2.0:1775773291.927614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664372096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:2.0:1775773291.927651:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c37800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664367360:192.168.201.118@tcp:400:kworker.0 00000100:00000400:2.0:1775773296.023240:0:404331:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773280/real 1775773280] req@ffff982971f97800 x1862032664369280/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773296.023266:0:404331:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982971f97800 x1862032664369280/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773296 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773296.023273:0:404331:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff98294542a800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1775773296.023283:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664369280:192.168.201.118@tcp:400:kworker.0 00000100:00000400:2.0:1775773297.047197:0:404331:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff982971f95f80 x1862032664370176/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773297.047233:0:404331:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982971f95f80 x1862032664370176/t0(0) o400->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 224/224 e 0 to 1 dl 1775773297 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773297.047241:0:404331:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff98294542a800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1775773297.047252:0:404331:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664370176:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773297.047270:0:404331:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1775773297.047280:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c36680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664372224:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773297.047367:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c34380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:14dd209b-d605-4024-a2b7-27ec435c0914:404331:1862032664372352:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773297.047405:0:404331:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:df0ee634-793c-4832-831b-03749b568504:404331:1862032664372608:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773301.655213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 1 seconds 00000100:00100000:2.0:1775773302.167283:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c34700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664372736:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773302.167399:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981c35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664372992:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773302.679233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 1 seconds 00000100:00000400:2.0:1775773303.703211:0:412860:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773262/real 1775773262] req@ffff9829745c7480 x1862032664365952/t0(0) o101->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 592/1328 e 0 to 1 dl 1775773303 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1775773303.703253:0:412860:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1775773303.703257:0:412860:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff982942b6f800: Connection to lustre-MDT0001 (at 192.168.201.118@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1775773303.703262:0:412860:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1775773303.703281:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773303.703451:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664373248:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773303.703556:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664373632:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773304.727203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773305.751255:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00000400:1.0:1775773306.263171:0:404330:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773265/real 1775773265] req@ffff9829745c7800 x1862032664366848/t0(0) o400->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773306.263217:0:404330:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9829745c7800 x1862032664366848/t0(0) o400->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773306 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773306.263227:0:404330:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1775773306.263231:0:404330:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9829493c7800: Connection to lustre-MDT0001 (at 192.168.201.118@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1775773306.263237:0:404330:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00000400:1.0:1775773306.263262:0:404330:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773265/real 1775773265] req@ffff9829745c4700 x1862032664366336/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773306.263278:0:404330:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9829745c4700 x1862032664366336/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773306 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773306.263284:0:404330:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b5a000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:1.0:1775773306.263304:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:df0ee634-793c-4832-831b-03749b568504:404330:1862032664366848:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773306.263309:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664366336:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773306.263447:0:404332:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664373888:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773306.263598:0:404332:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1775773306.263617:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:0:1862032664373760:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773306.775153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773307.287389:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773307.287406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775773307.287415:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664369152:192.168.201.118@tcp:400:kworker.0 00000100:00080000:1.0:1775773307.287419:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15064 00000100:00080000:1.0:1775773307.287438:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1775773307.287445:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773307.287486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982977d4c000 x1862032664374400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773307.287513:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287517:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773307.287519:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287522:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773307.287524:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773307.287527:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773307.287530:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15064 00000100:00080000:1.0:1775773307.287533:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1775773307.287537:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773307.287547:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773307.287564:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664374528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773307.287571:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287574:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773307.287575:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773307.287578:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15064 00000100:00080000:1.0:1775773307.287581:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1775773307.287584:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773307.287587:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773307.287594:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664374656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773307.287602:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287605:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773307.287608:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773307.287610:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1775773307.799157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773308.823157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773309.847305:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773310.871159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00000400:2.0:1775773311.383185:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773270/real 1775773270] req@ffff982981c34e00 x1862032664367232/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773311.383224:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c34e00 x1862032664367232/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773311 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773311.383235:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b5a000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1775773311.383254:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c34e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664367232:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773311.895169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773312.919140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773313.431158:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773303/real 1775773303] req@ffff982980f9f100 x1862032664373376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773313.431186:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664373376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773313 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773313.431199:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773313.431217:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773313.431220:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664373376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773313.431375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773313.943176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00000400:1.0:1775773316.503196:0:404330:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773275/real 1775773275] req@ffff9829745c4e00 x1862032664368000/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773316.503228:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773306/real 1775773306] req@ffff982980f9df80 x1862032664374272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773316.503240:0:404330:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9829745c4e00 x1862032664368000/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773316.503258:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664374272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773316.503261:0:404330:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b58000 already not connected (conn 6, was 5): CONNECTING 00000100:00000400:1.0:1775773316.503272:0:404330:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773275/real 1775773275] req@ffff982974b2f480 x1862032664368512/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773316.503297:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773306/real 1775773306] req@ffff982980f9e680 x1862032664374016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773316.503307:0:404330:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982974b2f480 x1862032664368512/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773316.503328:0:404330:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9829493c0000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1775773316.503328:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664374016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773316.503348:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829745c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:14dd209b-d605-4024-a2b7-27ec435c0914:404330:1862032664368000:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773316.503350:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773306/real 1775773306] req@ffff982980f9ed80 x1862032664374144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773316.503361:0:404330:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:df0ee634-793c-4832-831b-03749b568504:404330:1862032664368512:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773316.503393:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664374144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773316 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773316.503418:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773316.503443:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773316.503446:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664374272:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773316.503451:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773316.503456:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773316.503459:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664374016:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773316.503464:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773316.503468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503473:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775773316.503475:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775773316.503477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775773316.503480:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374144:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773316.503484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773316.503488:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773316.503496:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15078 00000100:00080000:1.0:1775773316.503505:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775773316.503510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773316.503525:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773316.503549:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664374912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773316.503572:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503576:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503578:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503582:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503585:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503589:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503592:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503595:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773316.503598:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775773316.503601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374912:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773316.503606:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15078 00000100:00080000:1.0:1775773316.503612:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775773316.503616:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773316.503623:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773316.503637:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664375040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773316.503645:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503648:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773316.503649:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773316.503651:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15078 00000100:00080000:1.0:1775773316.503655:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775773316.503657:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773316.503661:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773316.503668:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664375168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773316.503679:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503682:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503685:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503687:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503690:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503693:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773316.503695:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773316.503699:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1775773316.503699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664375040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773316.503724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664375168:192.168.201.118@tcp:38:kworker.0 00000100:00000400:2.0:1775773321.623195:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773280/real 1775773280] req@ffff982971f97480 x1862032664368896/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773321.623223:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982971f97480 x1862032664368896/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773321 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773321.623229:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b58000 already not connected (conn 7, was 5): CONNECTING 00000100:00000400:2.0:1775773321.623235:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773280/real 1775773280] req@ffff982971f96300 x1862032664369408/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773321.623242:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982971f96300 x1862032664369408/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773321 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773321.623247:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9829493c0000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1775773321.623257:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664368896:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773321.623261:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664369408:192.168.201.118@tcp:400:kworker.0 00000800:00000100:1.0:1775773322.135192:0:404320:0:(socklnd_cb.c:2376:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.201.118@tcp (192.168.201.118:988) timed out; the network or node may be down. 00000100:00000400:2.0:1775773322.647188:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff9829764b7100 x1862032664370304/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773322.647229:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9829764b7100 x1862032664370304/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773322 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773322.647236:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9829493c0000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1775773322.647242:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773281/real 1775773281] req@ffff982971f96680 x1862032664369920/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773322.647250:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982971f96680 x1862032664369920/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773322 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773322.647255:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b5a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1775773322.647265:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664370304:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773322.647269:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982971f96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664369920:192.168.201.118@tcp:400:kworker.0 00000100:00100000:1.0:1775773327.767170:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773307/real 1775773307] req@ffff982981fe0380 x1862032664374528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773327.767232:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664374528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773327.767263:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773307/real 1775773307] req@ffff982981fe3480 x1862032664374656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773327.767279:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664374656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773327.767303:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773307/real 1775773307] req@ffff982977d4c000 x1862032664374400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773327.767320:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982977d4c000 x1862032664374400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773327 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773327.767371:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773327.767396:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773327.767406:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374528:192.168.201.118@tcp:8:kworker.0 00000100:00080000:1.0:1775773327.767416:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773327.767447:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773327.767456:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374656:192.168.201.118@tcp:8:kworker.0 00000100:00080000:1.0:1775773327.767471:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773327.767493:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773327.767502:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982977d4c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664374400:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773327.767633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664375296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773327.767743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664375424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773327.767763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664375552:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773328.791171:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773313/real 1775773313] req@ffff982980f9ca80 x1862032664374784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773328.791215:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664374784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773328 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773328.791234:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773328.791258:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773328.791263:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773328.791506:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664375680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773331.863192:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773316/real 1775773316] req@ffff982981fe2a00 x1862032664375040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773331.863221:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664375040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773331.863233:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773316/real 1775773316] req@ffff982981fe2680 x1862032664375168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773331.863240:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664375168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773331.863247:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773316/real 1775773316] req@ffff982981fe2300 x1862032664374912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773331.863254:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664374912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773331 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773331.863265:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773331.863285:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773331.863288:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664375040:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773331.863292:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773331.863296:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773331.863298:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664375168:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773331.863301:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773331.863305:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773331.863307:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664374912:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773331.863331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863337:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775773331.863341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773331.863348:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773331.863355:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15089 00000100:00080000:2.0:1775773331.863367:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1775773331.863374:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773331.863386:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773331.863416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664375808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773331.863445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863448:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775773331.863450:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863452:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775773331.863454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863457:0:227016:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775773331.863458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773331.863462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773331.863465:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15089 00000100:00080000:2.0:1775773331.863468:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1775773331.863472:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773331.863476:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664375808:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775773331.863478:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773331.863487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664375936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773331.863494:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773331.863497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773331.863498:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773331.863501:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15089 00000100:00080000:2.0:1775773331.863504:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1775773331.863506:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773331.863510:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773331.863518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664376064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773331.863537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664375936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773331.863555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664376064:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773332.631288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1775773332.887184:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773291/real 1775773291] req@ffff982981c34a80 x1862032664371456/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773332.887227:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c34a80 x1862032664371456/t0(0) o400->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773332.887248:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b5a000 already not connected (conn 4, was 1): CONNECTING 00000100:00000400:2.0:1775773332.887258:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773291/real 1775773291] req@ffff982981c36a00 x1862032664371712/t0(0) o400->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773332.887276:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c36a00 x1862032664371712/t0(0) o400->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773332.887283:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9829493c5800 already not connected (conn 4, was 1): CONNECTING 00000100:00000400:2.0:1775773332.887295:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773291/real 1775773291] req@ffff982981c36300 x1862032664371328/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773332.887304:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c36300 x1862032664371328/t0(0) o400->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773332.887309:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff982945b58000 already not connected (conn 8, was 5): CONNECTING 00000100:00000400:2.0:1775773332.887312:0:404332:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775773291/real 1775773291] req@ffff982981c37480 x1862032664371584/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773332.887318:0:404332:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff982981c37480 x1862032664371584/t0(0) o400->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1775773332 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773332.887326:0:404332:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9829493c0000 already not connected (conn 4, was 1): CONNECTING 00000100:00100000:2.0:1775773332.887338:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c34a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664371456:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773332.887344:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c36a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664371712:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773332.887349:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c36300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:14dd209b-d605-4024-a2b7-27ec435c0914:404332:1862032664371328:192.168.201.118@tcp:400:kworker.0 00000100:00100000:2.0:1775773332.887354:0:404332:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981c37480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:df0ee634-793c-4832-831b-03749b568504:404332:1862032664371584:192.168.201.118@tcp:400:kworker.0 00010000:00010000:1.0:1775773335.703214:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773336.727246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773338.775315:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773339.799322:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773341.847231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773342.615242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773342.871258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773343.639173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773344.663413:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773344.919273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775773345.687232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773345.943286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775773346.199219:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.118:988 00000400:00000100:1.0:1775773346.199273:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775773346.199292:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199341:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199352:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199360:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199369:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199382:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773346.199397:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00080000:1.0:1775773346.199462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773346.199472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199480:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15100 00000100:00080000:1.0:1775773346.199496:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773346.199502:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773346.199533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664376192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.199564:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199569:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199573:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199576:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199579:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199582:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773346.199586:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199591:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15100 00000100:00080000:1.0:1775773346.199595:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773346.199597:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.199605:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773346.199617:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664376320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.199624:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199626:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773346.199628:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199631:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15100 00000100:00080000:1.0:1775773346.199634:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773346.199636:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.199639:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773346.199652:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664376448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.199659:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199661:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199663:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199666:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199674:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199681:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199686:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199692:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199695:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199699:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199705:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199717:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199788:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199794:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199800:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199812:0:374346:0:(pinger.c:247:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775773346.199818:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199824:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773346.199827:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199837:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15104 00000100:00080000:1.0:1775773346.199846:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1775773346.199854:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.199862:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773346.199875:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664376576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.199895:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199901:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773346.199903:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199906:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15104 00000100:00080000:1.0:1775773346.199908:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1775773346.199910:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.199913:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773346.199921:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664376704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.199942:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.199946:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773346.199948:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.199954:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15104 00000100:00080000:1.0:1775773346.199958:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1775773346.199964:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.199968:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773346.199977:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664376832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773346.200000:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773346.200005:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773346.200007:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773346.200012:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15101 00000100:00080000:1.0:1775773346.200018:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1775773346.200025:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773346.200030:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773346.200040:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664376960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773346.711181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773347.735191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773347.991241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775773348.759203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773349.015244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775773349.783209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773350.807173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773351.063222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773351.319306:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664377088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773351.319429:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773351.319463:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773351.319486:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773351.319524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773351.319557:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664377728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773351.319594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664377856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773351.831164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773352.855168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:2.0:1775773353.111241:0:408571:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 6/6 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1775773353.111294:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773353.111597:0:404330:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff982974b2ed80 x1862032664377984/t0(0) o103->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:17/18 lens 360/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773353.879182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773354.391430:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9ea00 x1862032664377216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391459:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664377216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391468:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391472:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391475:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773354.391480:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9fb80 x1862032664377344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664377344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391503:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773354.391509:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9f800 x1862032664377472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664377472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391531:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773354.391545:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9e300 x1862032664377088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391557:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664377088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391565:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664377088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773354.391597:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9dc00 x1862032664377856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664377856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664377856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773354.391651:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9d500 x1862032664377600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391663:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664377600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664377600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773354.391686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773351/real 1775773354] req@ffff982980f9c380 x1862032664377728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773354.391698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664377728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773354.391706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773354.391709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773354.391725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664377728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773354.903255:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773355.159237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773355.927167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773356.439281:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664378112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773356.439392:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773356.439429:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773356.439476:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773356.439523:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773356.439570:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664378752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773356.439616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664378880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773356.951182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:2.0:1775773357.463320:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.118:988 00000400:00000100:2.0:1775773357.463357:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773357.463384:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463423:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463435:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463447:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463456:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463469:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773357.463479:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773357.463492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9c000 x1862032664378624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664378624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773357.463549:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9c700 x1862032664378496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463562:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664378496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463570:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463574:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773357.463584:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9ce00 x1862032664378240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664378240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773357.463618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9d180 x1862032664378368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664378368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664378368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773357.463657:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9d880 x1862032664378112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664378112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664378112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773357.463709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9d500 x1862032664378880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664378880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664378880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773357.463743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773356/real 1775773357] req@ffff982980f9c380 x1862032664378752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773357.463753:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664378752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773357.463760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773357.463763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773357.463767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664378752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773357.975184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773358.231209:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773358.999154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773360.023211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773361.047188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773361.303236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773361.559305:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664379008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773361.559415:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773361.559447:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773361.559468:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773361.559526:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773361.559565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664379648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773361.559586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664379776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773362.071204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773363.095172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773364.119185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773364.375260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773364.631575:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9ea00 x1862032664379520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664379520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773364.631642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9ca80 x1862032664379648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664379648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664379648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773364.631683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9e300 x1862032664379136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664379136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773364.631718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9f800 x1862032664379264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631731:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664379264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773364.631756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9fb80 x1862032664379392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664379392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631777:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631780:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664379392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773364.631792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9dc00 x1862032664379008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664379008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631814:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664379008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773364.631874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773361/real 1775773364] req@ffff982980f9ed80 x1862032664379776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773364.631888:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664379776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773364.631896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773364.631899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773364.631902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664379776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773365.143173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773366.167194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773366.679359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664379904:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773366.679461:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.679512:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9e680 x1862032664379904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.679568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664379904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.679592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.679600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.679608:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664379904:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773366.679660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773366.679711:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.679731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9df80 x1862032664380032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.679744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664380032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.679753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.679756:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.679759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773366.679765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773366.679787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.679798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9f100 x1862032664380160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.679810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664380160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.679821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.679825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.679829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773366.679835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773366.679860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.679874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9f480 x1862032664380288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.679888:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664380288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.679897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.679900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.679904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773366.679911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773366.679935:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.679948:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9ed80 x1862032664380416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.679960:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664380416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.679989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.679993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.679997:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773366.680004:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664380544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773366.680028:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.680041:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9dc00 x1862032664380544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.680055:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664380544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.680104:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.680110:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.680114:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664380544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773366.680142:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664380672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773366.680174:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773366.680189:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773366/real 1775773366] req@ffff982980f9fb80 x1862032664380672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773366.680201:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664380672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773366.680210:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773366.680219:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773366.680222:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664380672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773367.191156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773367.447266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773368.215204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773369.239153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773370.263186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773370.519309:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773371.287173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773371.799265:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664380800:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773371.799326:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799359:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9f800 x1862032664380800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799418:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664380800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799441:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799449:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799454:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664380800:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773371.799477:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773371.799503:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9e300 x1862032664380928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664380928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799528:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799530:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664380928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773371.799536:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773371.799546:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9ca80 x1862032664381056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664381056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799568:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799571:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799573:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773371.799576:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773371.799592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799603:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9ea00 x1862032664381184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664381184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799623:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773371.799646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381312:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773371.799663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799671:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9c380 x1862032664381312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664381312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773371.799693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664381440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773371.799703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9d500 x1862032664381440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664381440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664381440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773371.799741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664381568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773371.799761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773371.799773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773371/real 1775773371] req@ffff982980f9d880 x1862032664381568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773371.799785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664381568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773371.799793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773371.799795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773371.799797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664381568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773372.311202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773373.335212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773373.591251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775773374.359180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773375.383202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773376.407202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773376.663293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773376.919280:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664381696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773376.919372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773376.919415:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773376.919452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773376.919485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773376.919504:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664382336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773376.919532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664382464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773377.431142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773378.455182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773379.479181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773379.735223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775773379.991320:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773379.991354:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773379.991372:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991422:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991431:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991440:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991450:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991461:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773379.991471:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773379.991479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2fb80 x1862032664382208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991519:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664382208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773379.991549:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2d880 x1862032664382336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d880 x1862032664382336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991600:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664382336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773379.991607:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2ce00 x1862032664381824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ce00 x1862032664381824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773379.991640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2d180 x1862032664381952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664381952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991661:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664381952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773379.991675:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2ca80 x1862032664382080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991687:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664382080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991695:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773379.991708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2df80 x1862032664381696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664381696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664381696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773379.991779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773376/real 1775773379] req@ffff982974b2e680 x1862032664382464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773379.991792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664382464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773379.991800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773379.991804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773379.991807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664382464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773380.503133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773381.527133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773382.039380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664382592:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773382.039481:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.039551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9d180 x1862032664382592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.039599:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664382592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.039621:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.039631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.039639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664382592:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773382.039678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773382.039752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.039770:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9ce00 x1862032664382720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.039784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664382720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.039794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.039799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.039803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773382.039810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773382.039833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.039845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9c700 x1862032664382848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.039858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664382848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.039874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.039878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.039882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773382.039889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773382.039912:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.039922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9c000 x1862032664382976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.039935:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664382976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.039943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.039947:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.039950:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664382976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773382.039958:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773382.039979:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.039990:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9d880 x1862032664383104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.040002:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664383104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.040011:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.040014:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.040018:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773382.040024:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664383232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773382.040043:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.040053:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9d500 x1862032664383232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.040112:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664383232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.040122:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.040126:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.040130:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664383232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773382.040154:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664383360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773382.040177:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773382.040189:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773382/real 1775773382] req@ffff982980f9c380 x1862032664383360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773382.040202:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664383360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773382.040210:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773382.040214:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773382.040217:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664383360:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773382.807315:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773385.879293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773387.159365:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664383488:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773387.159485:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159539:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9ea00 x1862032664383488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664383488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664383488:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773387.159660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773387.159687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159717:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9ca80 x1862032664383616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159753:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664383616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159776:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773387.159787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773387.159820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9e300 x1862032664383744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664383744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773387.159861:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773387.159876:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9f800 x1862032664383872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664383872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159899:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664383872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773387.159903:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773387.159915:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159921:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9fb80 x1862032664384000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664384000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159938:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159940:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159942:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773387.159946:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664384128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773387.159956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.159962:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9dc00 x1862032664384128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.159970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664384128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.159978:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.159982:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.159985:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664384128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773387.160007:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664384256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773387.160030:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773387.160040:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773387/real 1775773387] req@ffff982980f9ed80 x1862032664384256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773387.160049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664384256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773387.160054:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773387.160057:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773387.160060:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664384256:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773388.951252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773392.023273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773392.279311:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664384384:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773392.279397:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279442:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9f480 x1862032664384384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279468:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664384384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279492:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279502:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279507:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664384384:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773392.279548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773392.279568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279578:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9f100 x1862032664384512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664384512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279593:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279598:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773392.279603:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773392.279615:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9df80 x1862032664384640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279629:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664384640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773392.279642:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773392.279655:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279661:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9e680 x1862032664384768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664384768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279673:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279677:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773392.279680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773392.279691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9ed80 x1862032664384896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664384896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664384896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773392.279720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773392.279730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9dc00 x1862032664385024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664385024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279748:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773392.279755:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773392.279764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773392.279770:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773392/real 1775773392] req@ffff982980f9fb80 x1862032664385152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773392.279776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664385152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773392.279785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773392.279787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773392.279789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385152:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773395.095205:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773397.399245:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664385280:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773397.399338:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773397.399382:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773397.399402:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773397.399439:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773397.399460:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773397.399480:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664386048:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773398.167271:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773400.471631:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9ca80 x1862032664385536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664385536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773400.471704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9ea00 x1862032664385664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471718:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664385664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773400.471741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9c380 x1862032664385792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664385792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773400.471777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9d500 x1862032664385920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664385920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471802:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664385920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773400.471816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9f800 x1862032664385280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664385280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664385280:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773400.471848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9e300 x1862032664385408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664385408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471870:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664385408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773400.471911:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773397/real 1775773400] req@ffff982980f9d880 x1862032664386048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773400.471924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664386048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773400.471932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773400.471936:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773400.471939:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664386048:192.168.201.118@tcp:38:kworker.0 00010000:00010000:2.0:1775773401.239241:0:408571:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1775773401.239274:0:404833:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775773401.239286:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff982942b6f800 lock: 00000000cfd41984/0x12ae90f9442cc837 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x13c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x96e4bd9bff1c194d expref: -99 pid: 405510 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775773401.239370:0:404833:0:(vvp_io.c:1817:vvp_io_init()) [0x2000013a4:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775773401.239386:0:404833:0:(vvp_io.c:288:vvp_io_fini()) [0x2000013a4:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775773401.239466:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff982942b6f800 lock: 00000000cfd41984/0x12ae90f9442cc837 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x13c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x96e4bd9bff1c194d expref: -99 pid: 405510 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773401.239473:0:404833:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775773401.239505:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff982942b6f800 lock: 00000000cfd41984/0x12ae90f9442cc837 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x13c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x96e4bd9bff1c194d expref: -99 pid: 405510 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773401.239529:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773401.239532:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773401.239534:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773402.519427:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664386304:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773402.519540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.519600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2c380 x1862032664386304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.519644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664386304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.519684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.519700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.519711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664386304:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773402.519733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773402.519783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.519804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2f100 x1862032664386432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.519819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664386432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.519828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.519832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.519836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773402.519869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773402.519889:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.519900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2e680 x1862032664386560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.519912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664386560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.519921:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.519925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.519928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773402.519934:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773402.519955:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.519967:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2df80 x1862032664386688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.519980:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664386688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.519990:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.519995:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.519997:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773402.520004:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773402.520023:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.520032:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2ca80 x1862032664386816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.520045:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664386816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.520053:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.520056:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.520060:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664386816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773402.520119:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664386944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773402.520140:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.520152:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2d180 x1862032664386944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.520165:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664386944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.520174:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.520178:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.520181:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664386944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773402.520188:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773402.520209:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773402.520220:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773402/real 1775773402] req@ffff982974b2ce00 x1862032664387072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773402.520233:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ce00 x1862032664387072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773402.520243:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773402.520247:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773402.520251:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387072:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773404.311380:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773407.383308:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773407.639552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664387200:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773407.639638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.639690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9c000 x1862032664387200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.639731:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664387200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.639760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.639778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.639786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664387200:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773407.639806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773407.639839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.639855:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9c700 x1862032664387328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.639870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664387328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.639878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.639882:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.639885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773407.639909:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773407.639926:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.639935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9ce00 x1862032664387456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.639947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664387456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.639955:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.639958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.639961:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773407.639966:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773407.639995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.640005:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9d180 x1862032664387584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.640016:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664387584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.640024:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.640028:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.640031:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773407.640039:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773407.640116:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.640139:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9d880 x1862032664387712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.640154:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664387712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.640162:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.640166:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.640169:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664387712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773407.640175:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773407.640199:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.640208:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9e300 x1862032664387840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.640235:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664387840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.640266:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.640272:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.640278:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773407.640288:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773407.640323:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773407.640341:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773407/real 1775773407] req@ffff982980f9f800 x1862032664387968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773407.640354:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664387968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773407.640362:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773407.640365:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773407.640370:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664387968:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773410.455332:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773412.247153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773412.759441:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664388096:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773412.759559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.759609:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9d500 x1862032664388096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.759634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664388096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.759701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.759710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.759717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664388096:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773412.759732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773412.759771:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.759786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9c380 x1862032664388224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.759803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664388224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.759811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.759815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.759818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773412.759843:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773412.759859:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.759867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9ea00 x1862032664388352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.759877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664388352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.759883:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.759886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.759890:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773412.759895:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773412.759909:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.759917:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9ca80 x1862032664388480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.759926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664388480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.759932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.759935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.759937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773412.759942:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773412.759955:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.759963:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9fb80 x1862032664388608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.759972:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664388608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.759978:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.759980:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.759983:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664388608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773412.759987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664388736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773412.759999:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.760007:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9dc00 x1862032664388736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.760017:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664388736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.760040:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.760043:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.760045:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664388736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773412.760050:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664388864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773412.760067:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773412.760075:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773412/real 1775773412] req@ffff982980f9ed80 x1862032664388864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773412.760085:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664388864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773412.760196:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773412.760204:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773412.760209:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664388864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773413.271265:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773413.527385:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773414.295240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773414.551348:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773415.319186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773416.343288:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773416.599249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773417.367246:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773417.623314:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773417.879246:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879261:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773417.879274:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879287:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879304:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773417.879313:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773417.879369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664388992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773417.879416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879420:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773417.879422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879426:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879430:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773417.879433:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879445:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773417.879457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664389120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773417.879468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879474:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775773417.879475:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664388992:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773417.879476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879485:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879489:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773417.879493:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879500:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773417.879514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664389248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773417.879525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879529:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773417.879531:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879534:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879538:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773417.879542:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879547:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773417.879558:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664389376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775773417.879564:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773417.879571:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879574:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773417.879575:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879580:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879583:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773417.879586:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879589:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773417.879597:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664389504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773417.879604:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879607:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773417.879609:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879611:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879614:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773417.879618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b4e00 x1862032664388992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773417.879619:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879624:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773417.879638:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664389632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773417.879649:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664388992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773417.879651:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773417.879658:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773417.879662:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773417.879670:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15185 00000100:00080000:2.0:1775773417.879672:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773417.879675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773417.879675:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773417.879682:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775773417.879686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773417.879692:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664389760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773417.879694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664388992:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773417.879737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773417.879765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.879779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b7b80 x1862032664389120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.879793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664389120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.879803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.879807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.879811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773417.879818:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773417.879837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.879847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b5f80 x1862032664389248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.879861:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664389248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.879871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.879875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.879879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773417.879885:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773417.879904:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.879914:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b4700 x1862032664389376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.879927:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664389376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.879937:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.879941:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.879944:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773417.879951:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773417.879971:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.879994:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b5c00 x1862032664389504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.880009:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664389504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.880025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.880029:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.880033:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664389504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773417.880040:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664389632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773417.880062:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.880151:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b6d80 x1862032664389632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.880166:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664389632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.880175:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.880179:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.880183:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664389632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773417.880190:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664389760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773417.880211:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773417.880222:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773417/real 1775773417] req@ffff9829764b5180 x1862032664389760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773417.880236:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664389760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773417.880245:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773417.880258:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773417.880263:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664389760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773418.391215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773419.415168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773419.671299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773420.439140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773420.695330:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773421.463174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773422.487213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:2.0:1775773422.743263:0:408571:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 7/7 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1775773422.743290:0:408571:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/11 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1775773422.743331:0:404833:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 7 locks of 4 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1775773422.743345:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93200/0x12ae90f9442cd637 lrc: 2/0,0 mode: PR/PR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff1cfe32 expref: -99 pid: 410284 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1775773422.743425:0:404833:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a4:0xbcc:0x0]: losing layout lock 00010000:00010000:1.0:1775773422.743487:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93400/0x12ae90f9442cd622 lrc: 2/0,0 mode: CR/CR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff1cfdfa expref: -99 pid: 410283 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775773422.743534:0:404833:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a4:0xbcc:0x0]: losing layout lock 00010000:00010000:1.0:1775773422.743551:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93c00/0x12ae90f9442cd4c4 lrc: 2/0,0 mode: PR/PR res: [0x2000013a4:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff1c5552 expref: -99 pid: 408224 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743568:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982964842200/0x12ae90f9442cd278 lrc: 2/0,0 mode: PR/PR res: [0x2000013a4:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff1c4f16 expref: -99 pid: 407968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.743610:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982979ba7600/0x12ae90f9442cca91 lrc: 2/0,0 mode: PR/PR res: [0x2000013a4:0xa:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff1c22c3 expref: -99 pid: 406747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743721:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98294151d200/0x12ae90f9442cc94f lrc: 2/0,0 mode: PR/PR res: [0x2000013a4:0x5:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x96e4bd9bff1c1d13 expref: -99 pid: 406138 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743761:0:404833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff9829437cd000/0x12ae90f9442cc84c lrc: 2/0,0 mode: CR/CR res: [0x2000013a4:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x96e4bd9bff1c198c expref: -99 pid: 405531 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1775773422.743802:0:404833:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a4:0x3:0x0]: losing layout lock 00010000:00010000:1.0:1775773422.743839:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93200/0x12ae90f9442cd637 lrc: 1/0,0 mode: --/PR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1cfe32 expref: -99 pid: 410284 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.743848:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93400/0x12ae90f9442cd622 lrc: 1/0,0 mode: --/CR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1cfdfa expref: -99 pid: 410283 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773422.743854:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93c00/0x12ae90f9442cd4c4 lrc: 1/0,0 mode: --/PR res: [0x2000013a4:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c5552 expref: -99 pid: 408224 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743860:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982964842200/0x12ae90f9442cd278 lrc: 1/0,0 mode: --/PR res: [0x2000013a4:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c4f16 expref: -99 pid: 407968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.743866:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982979ba7600/0x12ae90f9442cca91 lrc: 1/0,0 mode: --/PR res: [0x2000013a4:0xa:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c22c3 expref: -99 pid: 406747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743872:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98294151d200/0x12ae90f9442cc94f lrc: 1/0,0 mode: --/PR res: [0x2000013a4:0x5:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c1d13 expref: -99 pid: 406138 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.743878:0:404833:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff9829437cd000/0x12ae90f9442cc84c lrc: 1/0,0 mode: --/CR res: [0x2000013a4:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1c198c expref: -99 pid: 405531 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.743883:0:404833:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 7 locks packed 00010000:00010000:1.0:1775773422.743909:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93200/0x12ae90f9442cd637 lrc: 0/0,0 mode: --/PR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1cfe32 expref: -99 pid: 410284 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.743931:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93400/0x12ae90f9442cd622 lrc: 0/0,0 mode: --/CR res: [0x2000013a4:0xbcc:0x0].0x0 bits 0xa/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1cfdfa expref: -99 pid: 410283 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775773422.743965:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98297fc93c00/0x12ae90f9442cd4c4 lrc: 0/0,0 mode: --/PR res: [0x2000013a4:0xe:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c5552 expref: -99 pid: 408224 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.744026:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982964842200/0x12ae90f9442cd278 lrc: 0/0,0 mode: --/PR res: [0x2000013a4:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c4f16 expref: -99 pid: 407968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.744046:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff982979ba7600/0x12ae90f9442cca91 lrc: 0/0,0 mode: --/PR res: [0x2000013a4:0xa:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c22c3 expref: -99 pid: 406747 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.744062:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff98294151d200/0x12ae90f9442cc94f lrc: 0/0,0 mode: --/PR res: [0x2000013a4:0x5:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x96e4bd9bff1c1d13 expref: -99 pid: 406138 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775773422.744139:0:404833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9829493c7800 lock: ffff9829437cd000/0x12ae90f9442cc84c lrc: 0/0,0 mode: --/CR res: [0x2000013a4:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x96e4bd9bff1c198c expref: -99 pid: 405531 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775773422.744162:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773422.744167:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773422.744172:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773422.999116:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999125:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773422.999129:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999138:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999150:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999155:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773422.999189:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664390144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773422.999213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773422.999218:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999221:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999225:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999228:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773422.999237:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773422.999261:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664390272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773422.999273:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999276:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773422.999279:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999284:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999289:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999294:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773422.999300:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773422.999330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664390144:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775773422.999339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664390400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773422.999354:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773422.999362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999367:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999373:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999379:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773422.999387:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773422.999404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664390528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775773422.999417:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773422.999434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773422.999441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999448:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999454:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773422.999466:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b6a00 x1862032664390144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773422.999471:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773422.999484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773422.999505:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664390144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773422.999511:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664390656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773422.999521:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999531:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773422.999538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664390144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773422.999540:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773422.999547:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775773422.999548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390272:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773422.999554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999562:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999570:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999577:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775773422.999579:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775773422.999591:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773422.999596:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b4a80 x1862032664390272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773422.999607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664390272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773422.999614:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664390784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773422.999624:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775773422.999642:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775773422.999644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390272:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773422.999652:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773422.999656:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773422.999664:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15190 00000100:00080000:2.0:1775773422.999671:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773422.999677:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773422.999680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390400:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775773422.999687:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773422.999713:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664390912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775773422.999718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773422.999753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b5180 x1862032664390400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773422.999768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664390400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773422.999789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773422.999800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773422.999808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773422.999833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773422.999843:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b6d80 x1862032664390528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773422.999854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664390528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773422.999861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773422.999867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773422.999872:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773422.999885:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773422.999892:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b5c00 x1862032664390656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773422.999919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664390656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773422.999925:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999927:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773422.999930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664390656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773422.999935:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664390784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773422.999948:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773422.999955:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773422] req@ffff9829764b4700 x1862032664390784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773422.999963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664390784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773422.999969:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773422.999971:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773422.999973:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664390784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773423.000009:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664390912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773423.000034:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773423.000042:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773422/real 1775773423] req@ffff9829764b5f80 x1862032664390912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773423.000051:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664390912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773423.000057:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773423.000059:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773423.000062:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664390912:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773425.815298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773428.119164:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119175:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773428.119180:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119191:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119204:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773428.119253:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664391040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119282:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773428.119284:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119287:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119290:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119294:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119304:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773428.119316:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664391168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119325:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119327:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773428.119329:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119332:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119334:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119337:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119340:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773428.119348:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664391296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119356:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773428.119360:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119363:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119365:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119367:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119371:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773428.119379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664391424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773428.119393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119396:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119400:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119402:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773428.119413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664391552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773428.119425:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119429:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119441:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119445:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773428.119455:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664391680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773428.119477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773428.119484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773428.119487:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773428.119496:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15195 00000100:00080000:1.0:1775773428.119500:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773428.119505:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773428.119508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773428.119514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664391808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773428.887324:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773429.655229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773430.679253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773431.703129:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773431.959206:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773432.727166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773433.239321:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664391936:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773433.239411:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773433.239452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773433.239501:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773433.239539:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773433.239577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664392576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773433.239591:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664392704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773433.751131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773434.775198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773435.031311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773435.799194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:2.0:1775773436.311312:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773436.311351:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773436.311366:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773436.311399:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773436.311419:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773436.311432:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773436.311441:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9df80 x1862032664392064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775773436.311444:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773436.311459:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773436.311471:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664392064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775773436.311475:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00080000:1.0:1775773436.311491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311498:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311503:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773436.311511:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9f100 x1862032664392192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311524:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664392192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311536:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311540:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773436.311547:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9e680 x1862032664391936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311560:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664391936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311571:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311574:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664391936:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773436.311602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9dc00 x1862032664392576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311624:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664392576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664392576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773436.311647:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9fb80 x1862032664392704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664392704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311683:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664392704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773436.311690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9f480 x1862032664392320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664392320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311719:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773436.311726:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773433/real 1775773436] req@ffff982980f9ed80 x1862032664392448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773436.311738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664392448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773436.311782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773436.311787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773436.311791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773436.823210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773437.847180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773438.103221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773438.359366:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664392832:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773438.359467:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359515:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2fb80 x1862032664392832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359562:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664392832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359583:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359591:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664392832:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773438.359634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773438.359660:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2f480 x1862032664392960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359687:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664392960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664392960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773438.359741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773438.359763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2e300 x1862032664393088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664393088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359805:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359808:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773438.359819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773438.359836:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2d500 x1862032664393216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664393216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359872:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773438.359878:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773438.359896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2ce00 x1862032664393344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359918:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ce00 x1862032664393344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773438.359939:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664393472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773438.359956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.359964:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2d180 x1862032664393472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.359975:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664393472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.359989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.359993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.359997:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664393472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773438.360003:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664393600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773438.360020:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773438.360030:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773438/real 1775773438] req@ffff982974b2ca80 x1862032664393600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773438.360042:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664393600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773438.360051:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773438.360054:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773438.360057:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664393600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773438.871212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773439.895180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773440.919212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773441.175345:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773441.943185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773442.967176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773443.479352:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664393728:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773443.479427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2df80 x1862032664393728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479508:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664393728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479534:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479554:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664393728:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773443.479583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773443.479650:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2e680 x1862032664393856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664393856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479699:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773443.479706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773443.479728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2f100 x1862032664393984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664393984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664393984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773443.479773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773443.479792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2c380 x1862032664394112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479816:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664394112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773443.479838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773443.479855:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2ca80 x1862032664394240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664394240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773443.479909:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664394368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773443.479927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479937:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2d180 x1862032664394368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.479949:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664394368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.479958:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.479962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.479965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664394368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773443.479972:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664394496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773443.479989:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773443.479998:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773443/real 1775773443] req@ffff982974b2ce00 x1862032664394496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773443.480011:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ce00 x1862032664394496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773443.480026:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773443.480029:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773443.480033:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664394496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773443.991198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773444.247262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773445.015161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773446.039175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773447.063174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773447.319230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773448.087215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773448.599335:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664394624:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773448.599407:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599439:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2d500 x1862032664394624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599465:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664394624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599508:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599518:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664394624:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773448.599568:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773448.599607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2e300 x1862032664394752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664394752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599641:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599646:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773448.599650:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773448.599662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2f480 x1862032664394880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664394880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664394880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773448.599697:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664395008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773448.599709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2fb80 x1862032664395008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664395008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664395008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773448.599742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664395136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773448.599761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2ce00 x1862032664395136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ce00 x1862032664395136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599797:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664395136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773448.599807:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664395264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773448.599826:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2d180 x1862032664395264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664395264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664395264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773448.599898:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664395392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773448.599919:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773448.599931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773448/real 1775773448] req@ffff982974b2ca80 x1862032664395392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773448.599940:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664395392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773448.599945:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773448.599947:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773448.599949:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664395392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773449.111139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773450.135226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773450.391224:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773451.159178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773452.183203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773453.207221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773453.463297:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773453.719235:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773453.719250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719260:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719273:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719280:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773453.719322:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664395520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773453.719375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719379:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719382:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719388:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773453.719414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664395648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773453.719433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719437:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773453.719484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664395776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719502:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773453.719503:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719515:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719519:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719527:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773453.719540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664395904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719554:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719563:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773453.719566:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719573:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719580:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719589:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719595:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773453.719610:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664396032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719623:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719628:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773453.719631:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719637:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719642:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719651:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719658:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773453.719675:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664396160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773453.719704:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773453.719710:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773453.719714:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773453.719721:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15221 00000100:00080000:1.0:1775773453.719729:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773453.719738:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773453.719748:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773453.719765:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664396288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773454.231163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773455.255187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773456.279188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773456.535280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773457.303184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773458.327190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773458.839164:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839172:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773458.839175:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839183:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839193:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839198:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773458.839231:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664396416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839259:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839264:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773458.839266:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839271:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839279:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839289:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773458.839306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664396544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839324:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773458.839326:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839341:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839345:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773458.839380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664396672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773458.839396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839400:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773458.839430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664396800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839436:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839440:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773458.839441:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839445:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839452:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773458.839465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664396928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839480:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773458.839481:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839485:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839489:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839496:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839501:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773458.839511:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664397056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773458.839525:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773458.839529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773458.839533:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773458.839541:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15226 00000100:00080000:1.0:1775773458.839545:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773458.839549:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773458.839554:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773458.839568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664397184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773459.607196:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773462.679232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773465.751333:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773468.823202:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773471.895251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773474.199428:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664399104:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773474.199512:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9ca80 x1862032664399104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199602:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664399104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199612:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664399104:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773474.199646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773474.199666:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9ea00 x1862032664399232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199685:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664399232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199694:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773474.199699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773474.199712:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9c380 x1862032664399360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664399360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773474.199762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773474.199785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9d500 x1862032664399488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664399488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773474.199838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773474.199854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199862:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9f800 x1862032664399616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664399616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664399616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773474.199882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664399744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773474.199892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9e300 x1862032664399744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664399744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199910:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199912:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199915:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664399744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773474.199921:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664399872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773474.199940:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773474.199952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773474/real 1775773474] req@ffff982980f9d880 x1862032664399872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773474.199966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664399872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773474.199974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773474.199977:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773474.199981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664399872:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773474.967222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773478.039242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773479.319279:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664400000:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773479.319340:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319374:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9d180 x1862032664400000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319402:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664400000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319415:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319420:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319426:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664400000:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773479.319458:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773479.319497:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319517:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9ce00 x1862032664400128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319531:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664400128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319551:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319555:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319558:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773479.319564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773479.319580:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9c700 x1862032664400256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319594:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664400256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319602:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319604:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773479.319607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773479.319618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319624:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9c000 x1862032664400384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664400384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319642:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773479.319654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773479.319664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9d880 x1862032664400512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319678:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664400512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319684:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664400512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773479.319692:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664400640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773479.319709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9e300 x1862032664400640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664400640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664400640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773479.319758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664400768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773479.319777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773479.319788:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773479/real 1775773479] req@ffff982980f9f800 x1862032664400768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773479.319801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664400768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773479.319810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773479.319814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773479.319817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664400768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773480.855255:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773481.879216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773482.903165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773483.927188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773484.439205:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773484.439216:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773484.439221:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439231:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439243:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773484.439251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773484.439287:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664400896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773484.439320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773484.439324:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773484.439327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439332:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439337:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773484.439341:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773484.439350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773484.439365:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664401024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773484.439377:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664400896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773484.439378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773484.439396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773484.439398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439404:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439409:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773484.439413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773484.439419:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773484.439435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664401152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775773484.439436:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773484.439449:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773484.439453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773484.439455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439460:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439466:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773484.439469:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b7b80 x1862032664400896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773484.439474:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773484.439485:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773484.439490:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664400896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773484.439502:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664401280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773484.439506:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439521:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664400896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773484.439522:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773484.439528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773484.439530:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439536:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439540:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773484.439544:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439546:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401024:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775773484.439552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773484.439568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664401408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775773484.439569:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b4e00 x1862032664401024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773484.439589:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664401024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773484.439592:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773484.439598:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773484.439598:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773484.439603:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773484.439605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401024:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773484.439613:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00100000:1.0:1775773484.439619:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401152:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773484.439620:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773484.439628:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773484.439634:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775773484.439646:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775773484.439651:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664401536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773484.439662:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775773484.439664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b7100 x1862032664401152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773484.439668:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773484.439672:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773484.439683:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15251 00000100:00080000:2.0:1775773484.439688:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773484.439692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664401152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773484.439693:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773484.439702:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775773484.439711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775773484.439719:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664401664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773484.439722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773484.439733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773484.439758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b5500 x1862032664401280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773484.439777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664401280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773484.439782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439784:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773484.439798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773484.439809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b6680 x1862032664401408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773484.439831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664401408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773484.439836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773484.439844:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664401536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773484.439854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b4000 x1862032664401536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773484.439868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664401536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773484.439873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439874:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664401536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773484.439880:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664401664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773484.439890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773484.439895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773484/real 1775773484] req@ffff9829764b7b80 x1862032664401664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773484.439902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664401664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773484.439929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773484.439932:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773484.439934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664401664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773484.951169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773485.975174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773486.999198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773489.559190:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773489.559204:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559214:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00080000:2.0:1775773489.559226:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559233:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773489.559270:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664401792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773489.559299:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559303:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773489.559305:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559310:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00080000:2.0:1775773489.559314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559318:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559328:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773489.559343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664401920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773489.559356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664401792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773489.559370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773489.559376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559381:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00080000:2.0:1775773489.559386:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559390:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773489.559408:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664402048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773489.559418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773489.559424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00080000:2.0:1775773489.559433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775773489.559437:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773489.559441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773489.559462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664402176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773489.559473:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773489.559480:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559486:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00100000:1.0:1775773489.559487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b5f80 x1862032664401792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773489.559492:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559500:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773489.559517:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664401792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773489.559522:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664402304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773489.559537:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773489.559543:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773489.559545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775773489.559551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775773489.559552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664401792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773489.559556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559563:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15257 00000100:00080000:2.0:1775773489.559568:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559572:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559577:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773489.559582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773489.559590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664402432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773489.559605:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773489.559609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773489.559611:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773489.559616:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15257 00000800:00000100:1.0:1775773489.559621:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773489.559624:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773489.559630:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773489.559637:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773489.559638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b4700 x1862032664401920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773489.559653:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664402560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773489.559670:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664401920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.559693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.559697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.559702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664401920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773489.559709:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773489.559730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773489.559740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b5c00 x1862032664402048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773489.559752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664402048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.559768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.559771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.559774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773489.559779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773489.559796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773489.559805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b6d80 x1862032664402176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773489.559830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664402176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.559840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.559843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.559847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773489.559853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773489.559870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773489.559880:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b5180 x1862032664402304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773489.559892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664402304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.559900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.559903:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.559907:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773489.559925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664402432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773489.559941:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773489.559950:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b4a80 x1862032664402432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773489.559962:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664402432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.559970:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.559973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.559976:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664402432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773489.559982:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664402560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773489.559997:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773489.560006:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773489/real 1775773489] req@ffff9829764b6a00 x1862032664402560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773489.560018:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664402560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773489.560026:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773489.560029:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773489.560032:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664402560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773494.679280:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664402688:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773494.679354:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679388:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9d500 x1862032664402688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679415:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664402688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679432:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679439:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679443:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664402688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773494.679476:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773494.679509:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679527:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9c380 x1862032664402816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679546:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664402816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679555:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679562:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773494.679581:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773494.679600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9ea00 x1862032664402944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664402944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664402944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773494.679633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664403072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773494.679645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9ca80 x1862032664403072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664403072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664403072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773494.679670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664403200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773494.679679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9ed80 x1862032664403200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664403200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664403200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773494.679720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664403328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773494.679755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9f480 x1862032664403328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664403328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664403328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773494.679784:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664403456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773494.679794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773494.679800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773494/real 1775773494] req@ffff982980f9fb80 x1862032664403456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773494.679812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664403456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773494.679821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773494.679824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773494.679827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664403456:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773499.799173:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799185:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773499.799190:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799199:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799212:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799218:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773499.799259:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664403584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799307:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799312:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773499.799315:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799321:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799327:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799346:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773499.799368:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664403712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773499.799388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799399:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799404:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773499.799426:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664403840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799460:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773499.799468:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799474:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799480:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799488:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799500:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773499.799524:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664403968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799561:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799566:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773499.799568:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799573:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799579:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799585:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773499.799617:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664404096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799648:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799654:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773499.799659:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799668:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799677:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799683:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799691:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773499.799709:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664404224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773499.799728:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773499.799733:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773499.799735:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773499.799741:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15267 00000100:00080000:1.0:1775773499.799748:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773499.799757:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773499.799763:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773499.799781:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664404352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919202:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773504.919206:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919218:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919230:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919236:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773504.919272:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664404480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773504.919312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919316:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919321:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919325:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919334:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773504.919349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664404608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773504.919373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919383:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773504.919406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664404736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919417:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773504.919423:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919431:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919436:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773504.919463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664404864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919482:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773504.919485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919495:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919500:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919505:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773504.919525:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664404992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919541:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919561:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773504.919565:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919576:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919581:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919586:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773504.919601:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664405120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773504.919615:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773504.919619:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773504.919621:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773504.919627:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15272 00000100:00080000:1.0:1775773504.919633:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773504.919639:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773504.919645:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773504.919659:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664405248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039173:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039183:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773510.039188:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039198:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039212:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039221:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773510.039271:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664405376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039311:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773510.039317:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039322:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039326:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039340:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773510.039362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664405504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773510.039406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039413:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039420:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039426:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039433:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773510.039476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664405632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773510.039495:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039499:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039505:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773510.039537:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664405760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039555:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039565:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773510.039570:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039575:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039583:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039590:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039597:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773510.039612:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664405888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039624:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039630:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773510.039635:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039640:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039645:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039648:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039653:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773510.039673:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664406016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773510.039692:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773510.039701:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773510.039705:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773510.039720:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15277 00000100:00080000:1.0:1775773510.039729:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773510.039743:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773510.039752:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773510.039770:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664406144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773515.159334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664406272:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773515.159423:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159460:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9dc00 x1862032664406272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159485:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664406272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159509:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159526:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664406272:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773515.159553:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773515.159573:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159582:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9e680 x1862032664406400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664406400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159599:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773515.159604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773515.159615:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9f100 x1862032664406528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159629:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664406528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159634:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159636:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773515.159642:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773515.159653:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9df80 x1862032664406656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664406656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159678:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159681:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773515.159690:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773515.159711:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9fb80 x1862032664406784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664406784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159743:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664406784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773515.159758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664406912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773515.159773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9f480 x1862032664406912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664406912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664406912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773515.159800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773515.159810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773515.159815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773515/real 1775773515] req@ffff982980f9ed80 x1862032664407040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773515.159822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664407040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773515.159827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773515.159829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773515.159832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773520.279357:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664407168:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773520.279417:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2df80 x1862032664407168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279479:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664407168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279503:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279509:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664407168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773520.279530:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773520.279550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2e680 x1862032664407296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664407296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279597:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279602:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279607:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773520.279611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773520.279632:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279647:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2f100 x1862032664407424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279663:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664407424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773520.279735:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773520.279757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2c380 x1862032664407552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664407552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773520.279788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773520.279798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2df80 x1862032664407680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664407680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279820:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664407680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773520.279823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773520.279833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2ca80 x1862032664407808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664407808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279850:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773520.279862:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773520.279872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773520.279877:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773520/real 1775773520] req@ffff982974b2d180 x1862032664407936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773520.279884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664407936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773520.279889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773520.279891:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773520.279893:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664407936:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775773525.399207:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773525.399222:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399233:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399247:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399254:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773525.399298:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664408064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399329:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773525.399330:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399334:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399337:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399339:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399349:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773525.399359:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664408192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773525.399371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399374:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399383:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399387:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773525.399398:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664408320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399407:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773525.399408:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399410:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399412:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399414:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399418:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773525.399425:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664408448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399434:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773525.399435:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399438:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399443:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399446:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773525.399452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664408576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773525.399462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399465:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399468:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399470:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773525.399481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664408704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773525.399489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773525.399492:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773525.399493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773525.399496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15292 00000100:00080000:1.0:1775773525.399499:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773525.399502:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773525.399505:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773525.399512:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664408832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775773526.167284:0:408571:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1775773526.167381:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773528.215245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773530.519215:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519228:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773530.519235:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519248:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519265:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519274:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773530.519325:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664409088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773530.519361:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519364:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519367:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773530.519389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664409216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773530.519400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519403:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519405:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519407:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519410:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773530.519417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664409344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773530.519430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519451:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773530.519469:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664409472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519494:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519498:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773530.519499:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519503:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519507:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519510:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773530.519526:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664409600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519536:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519540:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773530.519543:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519553:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519574:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519582:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773530.519598:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664409728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773530.519637:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773530.519649:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773530.519653:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773530.519657:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15298 00000100:00080000:1.0:1775773530.519664:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773530.519667:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773530.519674:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773530.519697:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664409856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773531.287225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773534.359282:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773535.639171:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639181:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773535.639185:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639195:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639207:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639213:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773535.639251:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664409984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639277:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639281:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773535.639283:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639287:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639291:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639294:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639304:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773535.639316:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664410112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639332:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773535.639337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639342:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639346:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639350:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639355:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773535.639367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664410240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639379:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773535.639380:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639384:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639388:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639395:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639400:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773535.639411:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664410368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639424:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773535.639427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639436:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639444:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639448:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639455:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773535.639471:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664410496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639494:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773535.639495:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639501:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639506:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639513:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639518:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773535.639532:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664410624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773535.639559:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773535.639566:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773535.639571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773535.639580:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15303 00000100:00080000:1.0:1775773535.639590:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773535.639597:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773535.639605:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773535.639621:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664410752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773537.431228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773540.503208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773543.575247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773546.647252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773549.719255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773550.743295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773552.535158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773552.791324:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773553.559183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773553.815309:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773554.583177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773555.607172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773555.863253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773556.119341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664413568:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773556.119403:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119435:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9ca80 x1862032664413568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119452:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664413568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119463:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119467:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119472:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664413568:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773556.119494:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773556.119514:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119525:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9ea00 x1862032664413696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119538:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664413696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119549:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119553:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773556.119559:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773556.119578:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9c380 x1862032664413824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119594:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664413824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773556.119607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773556.119618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9d500 x1862032664413952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664413952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664413952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773556.119643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773556.119679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9f800 x1862032664414080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664414080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773556.119706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664414208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773556.119717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9e300 x1862032664414208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664414208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664414208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773556.119751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664414336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773556.119761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773556.119767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773556/real 1775773556] req@ffff982980f9d880 x1862032664414336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773556.119774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664414336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773556.119779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773556.119781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773556.119783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664414336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773556.631177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773556.887236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773557.655187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773558.679209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773558.935275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773559.703172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773559.959254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773560.727177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773561.239200:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239210:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773561.239215:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00080000:2.0:1775773561.239238:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773561.239243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773561.239274:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664414464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773561.239297:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239300:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773561.239302:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00080000:2.0:1775773561.239311:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773561.239313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773561.239321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773561.239338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664414592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773561.239344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239347:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773561.239349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00080000:2.0:1775773561.239354:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773561.239357:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664414464:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775773561.239362:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773561.239377:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664414720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773561.239388:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773561.239394:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239398:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00080000:2.0:1775773561.239403:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775773561.239421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773561.239422:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773561.239448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773561.239459:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b6a00 x1862032664414464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773561.239462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664414848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773561.239496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773561.239503:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775773561.239507:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664414464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773561.239511:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00080000:2.0:1775773561.239520:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773561.239522:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239529:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775773561.239529:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239535:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664414464:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775773561.239540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773561.239543:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773561.239571:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664414976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773561.239582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775773561.239584:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773561.239590:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773561.239593:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239599:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15328 00000100:00100000:1.0:1775773561.239600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b4a80 x1862032664414592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773561.239607:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773561.239615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664414592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773561.239616:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775773561.239626:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775773561.239626:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775773561.239631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773561.239644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664415104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773561.239656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414720:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775773561.239657:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773561.239664:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773561.239666:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773561.239672:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15328 00000800:00000100:1.0:1775773561.239677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b5180 x1862032664414720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773561.239687:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773561.239697:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239700:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664414720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775773561.239708:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775773561.239719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775773561.239730:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664415232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773561.239733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773561.239753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b6d80 x1862032664414848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773561.239771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664414848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773561.239776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239781:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773561.239784:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773561.239794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b5c00 x1862032664414976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773561.239807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664414976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773561.239812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664414976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773561.239843:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664415104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773561.239856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239862:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b4700 x1862032664415104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773561.239870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664415104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773561.239875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664415104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773561.239882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664415232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773561.239891:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773561.239897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773561/real 1775773561] req@ffff9829764b5f80 x1862032664415232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773561.239903:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664415232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773561.239908:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773561.239910:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773561.239912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664415232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773561.751214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773562.007250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773562.775148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773563.031252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773563.799154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773564.823187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773565.079243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773565.847169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773566.103259:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773566.359352:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664415360:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773566.359508:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773566.359561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773566.359594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773566.359626:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773566.359654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773566.359670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773566.871185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773567.895160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773568.151290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773568.919193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773569.175291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775773569.431406:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773569.431479:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773569.431497:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431549:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431559:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431569:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431577:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431585:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773569.431593:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773569.431706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9f800 x1862032664416128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664416128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773569.431781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9c700 x1862032664415488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431795:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664415488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431808:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773569.431818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9ce00 x1862032664415616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664415616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773569.431854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9d180 x1862032664415744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664415744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431876:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431884:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773569.431908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9d880 x1862032664415872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431922:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664415872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431931:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431934:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664415872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773569.431945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9e300 x1862032664416000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431957:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664416000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.431966:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.431969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.431973:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773569.431980:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773566/real 1775773569] req@ffff982980f9c000 x1862032664415360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773569.431992:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664415360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773569.432000:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773569.432004:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773569.432007:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664415360:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773569.943136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773570.967179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773571.223267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773571.479286:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664416256:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773571.479362:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479424:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9d500 x1862032664416256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479446:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664416256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479471:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479479:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479489:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664416256:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773571.479524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773571.479559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479574:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9c380 x1862032664416384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664416384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479617:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773571.479633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773571.479661:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479678:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9ea00 x1862032664416512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664416512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773571.479708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773571.479729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9ca80 x1862032664416640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664416640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773571.479772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773571.479807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9ed80 x1862032664416768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664416768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664416768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773571.479863:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773571.479884:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9f480 x1862032664416896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664416896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479921:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479923:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664416896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773571.479929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773571.479946:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773571.479956:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773571/real 1775773571] req@ffff982980f9fb80 x1862032664417024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773571.479974:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664417024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773571.479981:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773571.479984:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773571.479988:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773571.991208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773572.247253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773573.015187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773574.039159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773575.063173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773575.319228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773576.087213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773576.599170:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773576.599179:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773576.599183:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599192:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599204:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599210:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773576.599247:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664417152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773576.599280:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773576.599285:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773576.599288:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599308:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599315:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599322:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773576.599335:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773576.599338:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664417152:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775773576.599355:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664417280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773576.599367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773576.599370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773576.599372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599376:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599381:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599385:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773576.599390:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773576.599401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664417408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773576.599413:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773576.599416:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775773576.599418:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773576.599419:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599425:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599430:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599434:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773576.599439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773576.599448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664417536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773576.599457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775773576.599462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b7b80 x1862032664417152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773576.599463:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773576.599467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599476:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599483:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773576.599494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773576.599498:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664417152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773576.599506:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664417664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773576.599515:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775773576.599519:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773576.599523:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775773576.599526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773576.599530:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775773576.599531:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664417152:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773576.599537:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599542:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773576.599559:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773576.599561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417280:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775773576.599591:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773576.599611:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664417792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773576.599623:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775773576.599624:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775773576.599631:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773576.599634:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773576.599640:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15344 00000100:00080000:2.0:1775773576.599645:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775773576.599646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b4000 x1862032664417280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775773576.599655:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773576.599666:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773576.599674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664417280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.599687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775773576.599687:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664417920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773576.599697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.599706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773576.599721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773576.599745:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773576.599755:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b6680 x1862032664417408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773576.599783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664417408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.599789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773576.599792:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.599794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773576.599799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773576.599814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773576.599822:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b5500 x1862032664417536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773576.599855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664417536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.599863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773576.599865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.599868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773576.599873:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417664:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773576.599889:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773576.599897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b7100 x1862032664417664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773576.599908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664417664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.599915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773576.599917:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.599920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664417664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773576.599953:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773576.599970:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773576.599979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b4e00 x1862032664417792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773576.600007:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664417792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.600019:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773576.600022:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.600024:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773576.600029:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773576.600042:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773576.600049:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773576/real 1775773576] req@ffff9829764b7b80 x1862032664417920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773576.600058:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664417920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773576.600066:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773576.600071:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773576.600075:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664417920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773577.112139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773578.135142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773578.391333:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773579.159190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773580.183181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773581.463246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773581.719385:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664418048:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773581.719463:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9df80 x1862032664418048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664418048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719578:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664418048:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773581.719595:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773581.719622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9f100 x1862032664418176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719649:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664418176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773581.719693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773581.719716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9e680 x1862032664418304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664418304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773581.719769:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773581.719803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9dc00 x1862032664418432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664418432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773581.719851:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773581.719872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9fb80 x1862032664418560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719896:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664418560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719911:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664418560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773581.719917:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664418688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773581.719939:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.719949:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9f480 x1862032664418688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.719962:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664418688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.719971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.719974:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.719978:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664418688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773581.719983:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664418816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773581.720003:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773581.720013:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773581/real 1775773581] req@ffff982980f9ed80 x1862032664418816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773581.720025:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664418816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773581.720033:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773581.720036:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773581.720039:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664418816:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773584.535274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773586.839536:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664418944:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773586.839669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.839735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9ca80 x1862032664418944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.839760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ca80 x1862032664418944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.839773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.839779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.839783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664418944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773586.839791:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773586.839815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.839827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9ea00 x1862032664419072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.839839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664419072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.839847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.839851:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.839854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773586.839879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773586.839897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.839908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9c380 x1862032664419200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.839927:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664419200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.839936:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.839939:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.839943:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773586.839949:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773586.839968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.839976:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9d500 x1862032664419328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.839984:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664419328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.839989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.839991:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.839993:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773586.839997:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773586.840008:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.840014:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9c000 x1862032664419456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.840020:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664419456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.840030:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.840032:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.840034:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773586.840038:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664419584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773586.840047:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.840052:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9e300 x1862032664419584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.840059:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664419584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.840064:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.840066:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.840068:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664419584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773586.840071:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664419712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773586.840128:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773586.840138:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773586/real 1775773586] req@ffff982980f9d880 x1862032664419712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773586.840146:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664419712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773586.840151:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773586.840153:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773586.840156:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664419712:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773587.607276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773590.679242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773591.959415:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664419840:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773591.959482:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959516:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2f100 x1862032664419840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664419840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959576:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664419840:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773591.959614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773591.959648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2c380 x1862032664419968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959685:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664419968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959695:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664419968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773591.959709:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773591.959732:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2df80 x1862032664420096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664420096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773591.959792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773591.959820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2e680 x1862032664420224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959844:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664420224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773591.959866:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773591.959887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959898:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2ca80 x1862032664420352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664420352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773591.959924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664420480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773591.959936:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959942:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2d180 x1862032664420480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959950:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664420480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.959955:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.959956:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.959958:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664420480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773591.959971:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664420608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773591.959982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773591.959988:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773591/real 1775773591] req@ffff982974b2d500 x1862032664420608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773591.959995:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664420608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773591.960004:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773591.960006:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773591.960008:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664420608:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773593.751202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773596.823293:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773597.079234:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079248:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773597.079254:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079266:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079281:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079290:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773597.079338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664420736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773597.079373:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079377:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079382:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079386:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079395:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773597.079408:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664420864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773597.079423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079426:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079429:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079432:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079436:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773597.079445:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664420992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773597.079465:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775773597.079467:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664420736:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775773597.079473:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079486:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079490:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773597.079507:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664421120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079515:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079518:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773597.079520:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079523:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079526:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079530:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079534:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773597.079543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664421248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079550:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079553:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773597.079554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079557:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079568:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079571:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775773597.079584:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775773597.079585:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664421376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773597.079597:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773597.079600:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773597.079602:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773597.079607:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15364 00000100:00080000:2.0:1775773597.079611:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773597.079614:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773597.079617:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773597.079623:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664421504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773597.079672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b5f80 x1862032664420736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.079710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664420736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.079726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.079732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.079738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664420736:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773597.079750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773597.079785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.079825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b4700 x1862032664420864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.079839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664420864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.079848:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.079852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.079855:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773597.079888:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773597.079908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.079918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b5c00 x1862032664420992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.079929:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664420992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.079968:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.079972:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.079975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664420992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773597.079987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664421120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773597.080008:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.080018:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b6d80 x1862032664421120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.080029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664421120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.080037:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.080040:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.080043:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664421120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773597.080049:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664421248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773597.080111:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.080121:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b5180 x1862032664421248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.080133:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664421248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.080140:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.080144:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.080146:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664421248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773597.080153:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664421376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773597.080171:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.080181:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b4a80 x1862032664421376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.080193:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664421376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.080202:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.080205:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.080209:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664421376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773597.080215:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664421504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773597.080234:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773597.080244:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773597/real 1775773597] req@ffff9829764b6a00 x1862032664421504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773597.080265:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664421504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773597.080286:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773597.080290:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773597.080294:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664421504:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773599.895279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773602.199220:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199231:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773602.199236:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199248:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199308:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199316:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773602.199363:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664421632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773602.199412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199416:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199418:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199421:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773602.199440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664421760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199448:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199451:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773602.199452:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199455:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199457:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199459:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773602.199478:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664421888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199484:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773602.199489:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199493:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199495:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199498:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199504:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773602.199519:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664422016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199530:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199545:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773602.199547:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199552:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199559:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199568:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773602.199601:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664422144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199610:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199613:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773602.199616:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199619:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199623:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199628:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199633:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773602.199640:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664422272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773602.199647:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773602.199649:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773602.199654:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773602.199661:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15369 00000100:00080000:1.0:1775773602.199665:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773602.199668:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773602.199674:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773602.199687:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664422400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773602.967225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773606.039263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773609.111212:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773610.135249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773612.183270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773612.439199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439207:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773612.439211:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439219:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439230:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439236:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773612.439272:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664423424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439296:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439298:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773612.439300:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439303:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439305:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439307:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439316:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773612.439325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664423552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773612.439336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439343:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773612.439361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664423680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773612.439372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439374:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439376:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439378:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439382:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773612.439388:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664423808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439401:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773612.439410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439414:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773612.439433:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664423936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439450:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773612.439453:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439458:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439464:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773612.439475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664424064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773612.439494:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773612.439499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773612.439502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773612.439509:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15379 00000100:00080000:2.0:1775773612.439514:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773612.439517:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773612.439523:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773612.439534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664424192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773613.207189:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773614.231319:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773615.255239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773616.279331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773617.303300:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773617.559376:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664424320:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775773617.560648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9d180 x1862032664424320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560706:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664424320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664424320:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775773617.560729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773617.560747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9ce00 x1862032664424448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560765:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664424448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773617.560810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773617.560833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9c700 x1862032664424576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664424576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560869:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560872:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773617.560879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773617.560894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560901:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9f800 x1862032664424704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664424704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560913:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773617.560922:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775773617.560934:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560939:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9d880 x1862032664424832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560946:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664424832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560952:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560953:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560955:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664424832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775773617.560960:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664424960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773617.560970:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.560976:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9e300 x1862032664424960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.560984:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664424960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.560989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.560990:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.560992:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664424960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775773617.560996:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664425088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775773617.561006:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775773617.561012:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775773617/real 1775773617] req@ffff982980f9c000 x1862032664425088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775773617.561020:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664425088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775773642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775773617.561025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775773617.561027:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775773617.561029:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664425088:192.168.201.118@tcp:38:kworker.0 00010000:00010000:1.0:1775773618.327301:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773619.351278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773621.399276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773622.423274:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773622.679281:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679293:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773622.679300:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679312:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679328:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679336:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773622.679384:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664425216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.679427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773622.679434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679440:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679444:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679451:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.679464:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773622.679509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664425344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.679557:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679564:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773622.679567:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679576:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679585:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679593:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.679602:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773622.679629:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664425472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.679650:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679657:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773622.679660:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679672:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679683:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679695:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.679705:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773622.679731:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664425600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.679761:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679771:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773622.679774:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679783:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679791:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679804:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.679815:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773622.679840:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664425728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.679887:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.679899:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773622.679907:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.679921:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.679935:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.679948:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.679959:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773622.679988:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664425856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773622.680034:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773622.680045:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773622.680050:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773622.680065:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15390 00000100:00080000:1.0:1775773622.680073:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773622.680080:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773622.680147:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773622.680175:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664425984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773624.474049:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773625.495270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773627.543292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773627.799146:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.799167:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773627.799171:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800517:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800535:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800569:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773627.800608:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664426112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.800633:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.800637:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773627.800639:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800643:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800647:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800650:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.800658:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773627.800675:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664426240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.800696:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.800701:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773627.800703:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800708:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800713:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800717:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.800724:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773627.800738:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664426368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.800754:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.800758:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773627.800760:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800766:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800771:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800788:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.800800:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773627.800817:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664426496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.800843:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.800847:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773627.800850:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800855:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800861:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800868:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.800875:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773627.800892:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664426624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.800922:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.800942:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773627.800947:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.800955:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.800962:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.800971:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.800977:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773627.800993:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664426752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773627.801012:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773627.801018:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773627.801021:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773627.801030:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15395 00000100:00080000:1.0:1775773627.801040:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773627.801047:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773627.801053:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773627.801085:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664426880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773628.567371:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773630.615251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773631.640506:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773633.687283:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773634.711384:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775773635.991726:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775773635.991808:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775773635.991850:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993729:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993754:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993763:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993778:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993785:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773635.993790:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773636.503291:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773636.759301:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773637.527169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773637.783331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773638.551253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773639.575220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773639.831326:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773640.599264:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773640.855444:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773641.623187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773642.647247:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773642.903263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773643.159290:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159307:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773643.159313:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159329:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159346:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159373:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773643.159445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664428800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159508:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159514:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773643.159517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159530:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159536:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159543:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159577:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773643.159594:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664428928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159606:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773643.159611:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159640:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159649:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159667:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159675:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773643.159711:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664429056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159720:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159723:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773643.159725:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159730:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159734:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159738:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159743:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773643.159752:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664429184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159776:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159779:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773643.159780:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159782:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159785:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159787:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159790:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773643.159796:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664429312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159802:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159804:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773643.159805:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159807:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159826:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159835:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159839:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773643.159855:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664429440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773643.159864:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773643.159866:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773643.159867:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773643.159870:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15410 00000100:00080000:1.0:1775773643.159872:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773643.159874:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773643.159878:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773643.159886:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664429568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773643.671219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773643.927387:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773644.695195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773645.975249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773646.999379:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773648.279214:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773648.279250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279262:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279278:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279286:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773648.279336:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664429696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279372:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773648.279374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279381:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279384:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773648.279412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664429824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773648.279430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279439:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279444:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279451:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773648.279470:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664429952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279479:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279482:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773648.279483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279486:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279490:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279492:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773648.279508:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664430080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279515:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279518:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773648.279519:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279522:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279524:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279527:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279529:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773648.279540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664430208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279549:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279553:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773648.279555:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279558:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279561:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279564:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773648.279574:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664430336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773648.279580:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773648.279583:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773648.279584:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773648.279587:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15415 00000100:00080000:2.0:1775773648.279591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773648.279597:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773648.279604:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773648.279617:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664430464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773649.047278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773650.071357:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773652.119308:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773653.143254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773655.191263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773656.215276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773658.263304:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773659.287277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773661.335244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773662.359232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773664.407322:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773665.431284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773667.223234:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773667.479230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773668.247177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773668.503277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773669.271202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773670.295184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773670.551224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773671.319166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773672.343193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773673.367169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773673.623227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773674.391185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773675.415177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773676.439153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773676.695248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773677.463181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773678.487161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773679.511203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773679.767245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773680.535171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773681.559198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773682.583205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773682.839239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773683.607185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773684.631170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773685.655169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773685.911300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775773686.679190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773687.703233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773688.727174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773689.239179:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239191:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773689.239196:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239207:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239219:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239225:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773689.239261:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664436864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239292:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773689.239295:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239300:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239304:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239309:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239332:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773689.239351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664436992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773689.239372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239376:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773689.239425:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664437120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773689.239450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239468:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773689.239503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664437248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239526:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773689.239529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239536:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239544:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239549:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239555:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773689.239571:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664437376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239598:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773689.239610:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239619:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239627:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239636:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239645:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773689.239659:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664437504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773689.239672:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773689.239676:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773689.239679:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773689.239684:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15456 00000100:00080000:1.0:1775773689.239691:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773689.239700:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773689.239707:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773689.239726:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664437632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773689.751179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773690.775184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773691.799183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773692.823182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773693.847167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773694.359260:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773694.359280:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359300:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359319:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359328:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773694.359379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664437760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773694.359430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359440:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359446:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773694.359474:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664437888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773694.359497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359502:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359506:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359516:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773694.359528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664438016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359539:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359543:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773694.359544:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359604:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359612:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359622:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359632:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773694.359650:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664438144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359666:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359671:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773694.359673:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359679:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359687:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359694:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359701:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773694.359723:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664438272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359743:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359750:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773694.359754:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359762:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359767:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359771:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359777:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773694.359796:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664438400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773694.359827:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773694.359836:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773694.359839:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773694.359845:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15461 00000100:00080000:2.0:1775773694.359851:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773694.359859:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773694.359866:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773694.359882:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664438528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773694.871237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773695.895169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773696.919168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773697.943235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773698.967193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773699.991159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773701.015255:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773702.039189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:2.0:1775773702.551486:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773702.551541:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773702.551564:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551616:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551629:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551640:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551653:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551666:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773702.551682:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773703.063201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773704.087214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773705.111223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773706.135226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773707.159248:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773708.183222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773709.207213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773710.231233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773711.255166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773712.279218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773713.303172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773714.327214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773715.351150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773716.375209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773717.399166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773718.423164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773719.447153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773720.471174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773721.495162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773722.519240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773723.543161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773724.567154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773725.079229:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079238:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773725.079242:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079251:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079262:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079267:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773725.079298:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664443136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079325:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079328:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773725.079333:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079336:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079339:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079342:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773725.079363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664443264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773725.079380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079386:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079388:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773725.079400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664443392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079407:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773725.079410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079412:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079424:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079431:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773725.079448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664443520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773725.079459:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079464:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773725.079481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664443648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773725.079500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079514:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079524:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079535:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773725.079553:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664443776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773725.079568:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773725.079574:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773725.079577:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773725.079581:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15492 00000100:00080000:1.0:1775773725.079585:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773725.079591:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773725.079599:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773725.079611:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664443904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773725.591168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773726.615179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773727.639195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773728.663195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773729.687211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773730.711170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773731.735174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773732.759177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773733.783176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773740.439198:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439209:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773740.439214:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439225:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439238:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439244:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773740.439280:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664445824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439304:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439307:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773740.439308:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439311:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439317:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439324:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773740.439332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664445952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439341:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773740.439342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773740.439358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664446080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439364:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773740.439376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439384:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439387:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439389:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773740.439402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664446208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773740.439414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439443:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439447:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773740.439462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664446336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439472:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773740.439474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439477:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439480:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439482:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439485:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773740.439493:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664446464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773740.439524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773740.439527:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773740.439528:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773740.439532:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15507 00000100:00080000:1.0:1775773740.439536:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773740.439543:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773740.439550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773740.439559:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664446592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559177:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559186:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773745.559190:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559198:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559209:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559215:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773745.559248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664446720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559272:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773745.559276:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559279:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559284:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559293:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773745.559301:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664446848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559309:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559311:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773745.559312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559315:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559317:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559320:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559323:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773745.559350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664446976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559361:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773745.559378:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559382:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559387:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559391:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773745.559410:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664447104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773745.559430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559433:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559437:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559440:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773745.559453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664447232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773745.559462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559465:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559468:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559473:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559479:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773745.559489:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664447360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773745.559495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773745.559497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773745.559498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773745.559504:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15513 00000100:00080000:1.0:1775773745.559509:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773745.559525:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773745.559532:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773745.559542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664447488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039175:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039182:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773766.039186:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039195:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039205:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773766.039248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664450304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039276:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773766.039284:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039289:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039293:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039297:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773766.039323:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664450432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773766.039335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039338:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039343:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773766.039353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664450560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773766.039384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039388:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039395:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039401:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773766.039413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664450688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773766.039428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039431:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039435:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039443:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773766.039459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664450816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773766.039477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039496:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039502:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039516:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773766.039532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664450944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773766.039545:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773766.039550:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773766.039555:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773766.039561:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15533 00000100:00080000:1.0:1775773766.039566:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773766.039571:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773766.039577:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773766.039591:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664451072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773768.855292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775773776.279205:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279214:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773776.279220:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279237:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279248:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279254:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773776.279293:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664452096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773776.279325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773776.279354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664452224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279362:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279366:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773776.279368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279373:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279383:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773776.279393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664452352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279402:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279405:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773776.279406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279410:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279412:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279415:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279420:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773776.279429:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664452480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773776.279441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279443:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279446:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279448:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279452:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773776.279464:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664452608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773776.279477:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279481:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279488:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279492:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773776.279503:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664452736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773776.279512:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773776.279521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773776.279526:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773776.279536:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15543 00000100:00080000:1.0:1775773776.279540:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773776.279554:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773776.279562:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773776.279576:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664452864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773796.247178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773797.271153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773798.295223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773799.319193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773800.343174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773801.367171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773801.879178:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879189:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773801.879194:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879203:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879213:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879218:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773801.879253:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664456576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879276:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879280:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773801.879281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879286:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879289:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879296:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773801.879305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664456704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879312:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879314:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773801.879315:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879318:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879322:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879325:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773801.879334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664456832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773801.879359:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879361:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879364:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879367:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879372:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773801.879381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664456960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773801.879391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879393:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879395:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879397:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879400:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773801.879408:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664457088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879415:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773801.879421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879425:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879427:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879429:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879432:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773801.879440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664457216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773801.879445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773801.879448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773801.879450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773801.879453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15569 00000100:00080000:1.0:1775773801.879457:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773801.879460:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773801.879463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773801.879471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664457344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773802.391228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773806.999186:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999194:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773806.999199:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999208:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999218:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999223:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773806.999270:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664457472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999305:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773806.999307:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999316:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999321:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999330:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773806.999345:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664457600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773806.999364:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999367:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999371:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999373:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999376:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773806.999385:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664457728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999391:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773806.999395:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999398:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999400:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999402:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773806.999417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664457856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773806.999428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999434:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999436:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773806.999448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664457984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999476:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773806.999488:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999491:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999494:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999497:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999502:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773806.999514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664458112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773806.999530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773806.999536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773806.999539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773806.999542:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15574 00000100:00080000:1.0:1775773806.999545:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773806.999548:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773806.999550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773806.999559:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664458240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119261:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119271:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773812.119277:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119289:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119301:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119308:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773812.119356:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664458368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773812.119388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119392:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119395:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119397:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773812.119416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664458496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773812.119428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119432:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119434:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119437:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119442:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773812.119449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664458624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119458:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773812.119463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119468:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119471:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119478:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773812.119502:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664458752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119511:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119514:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773812.119516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119519:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119521:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119524:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119529:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773812.119536:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664458880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119543:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773812.119548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119556:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119561:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119565:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119569:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773812.119580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664459008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773812.119588:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773812.119591:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773812.119594:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773812.119598:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15579 00000100:00080000:1.0:1775773812.119608:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773812.119618:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773812.119632:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773812.119644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664459136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239169:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239177:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773817.239182:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239200:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239206:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773817.239245:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664459264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773817.239293:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239297:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239301:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239305:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239315:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773817.239327:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664459392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239338:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773817.239346:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239349:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239352:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239358:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773817.239366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664459520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239373:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773817.239376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239379:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239382:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239384:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239388:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773817.239396:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664459648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773817.239413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239417:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239420:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239423:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773817.239434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664459776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773817.239451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239458:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239463:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773817.239479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664459904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773817.239503:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773817.239506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773817.239507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773817.239511:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15584 00000100:00080000:1.0:1775773817.239514:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773817.239516:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773817.239519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773817.239528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664460032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773824.919180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773825.943203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773826.967169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773827.479142:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479162:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773827.479167:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479175:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479187:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479192:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773827.479226:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664461056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479252:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479255:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773827.479257:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479260:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479263:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479265:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479272:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773827.479282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664461184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479290:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773827.479294:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479297:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479308:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773827.479321:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664461312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773827.479340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479349:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479354:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479363:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479367:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773827.479379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664461440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773827.479391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479394:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479397:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479399:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773827.479411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664461568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773827.479426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479430:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479449:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773827.479471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664461696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773827.479481:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773827.479486:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773827.479490:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773827.479503:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15594 00000100:00080000:1.0:1775773827.479507:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773827.479509:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773827.479521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773827.479534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664461824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773827.991155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773829.015177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773830.039270:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773831.063204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773832.087187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773833.111180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773834.135145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773835.159191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:2.0:1775773835.671333:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773835.671371:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773835.671386:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671418:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671425:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671430:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671463:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671473:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773835.671484:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773836.183153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773837.207131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773838.231224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773839.255195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773840.279181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773841.303153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773842.327184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773843.351145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773844.375216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773845.399169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773846.423241:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773847.447161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773847.959135:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959144:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773847.959148:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959160:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959171:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959178:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773847.959213:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664464640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959240:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773847.959246:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959251:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959254:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959259:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959269:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773847.959282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664464768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959320:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773847.959322:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959336:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773847.959355:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664464896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773847.959376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959385:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959388:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773847.959404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664465024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773847.959430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959440:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959444:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959449:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773847.959472:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664465152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959505:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959510:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773847.959514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959521:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959525:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959530:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959535:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773847.959563:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664465280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773847.959577:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773847.959586:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773847.959589:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773847.959597:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15615 00000100:00080000:2.0:1775773847.959601:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773847.959610:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773847.959618:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773847.959633:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664465408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773848.471204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773849.495162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773850.519218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773851.543170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773852.567197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773853.079204:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079213:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773853.079217:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079225:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079236:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773853.079275:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664465536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079306:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773853.079312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079321:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079325:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079335:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773853.079348:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664465664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079359:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079364:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773853.079368:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079372:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079374:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079376:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079380:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773853.079390:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664465792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773853.079401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079408:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773853.079417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664465920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773853.079427:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079432:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079437:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079440:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773853.079469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664466048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079481:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773853.079483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079490:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079495:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079500:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773853.079517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664466176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773853.079533:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773853.079539:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773853.079543:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773853.079546:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15620 00000100:00080000:2.0:1775773853.079548:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773853.079551:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773853.079553:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773853.079560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664466304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773853.591131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773854.615147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773855.639156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773856.663176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773857.687130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773858.199186:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199195:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773858.199200:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199210:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199222:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199229:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773858.199267:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664466432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199292:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199296:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773858.199298:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199302:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199306:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199309:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199319:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773858.199331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664466560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199342:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773858.199348:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199356:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199360:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199394:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773858.199414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664466688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199432:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773858.199435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199440:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199444:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199449:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773858.199464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664466816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199479:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773858.199481:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199486:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199500:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199505:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773858.199526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664466944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199541:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199544:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773858.199546:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199552:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199557:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199562:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199568:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773858.199582:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664467072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773858.199598:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773858.199609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773858.199611:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773858.199616:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15625 00000100:00080000:1.0:1775773858.199626:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773858.199636:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773858.199644:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773858.199659:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664467200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773858.711228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773859.735180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773860.759261:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773861.783186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773862.807213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773863.319176:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319185:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773863.319190:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319199:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319210:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319216:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773863.319250:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664467328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319305:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773863.319307:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319310:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319313:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319316:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319324:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773863.319337:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664467456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319348:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773863.319351:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319354:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319357:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319359:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319362:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773863.319371:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664467584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773863.319386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319388:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319391:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319392:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319396:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773863.319406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664467712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319413:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319417:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773863.319418:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319421:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319423:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319426:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319429:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773863.319438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664467840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319472:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773863.319477:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319481:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319487:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319494:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773863.319509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664467968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773863.319528:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773863.319530:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773863.319531:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773863.319534:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15630 00000100:00080000:1.0:1775773863.319536:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773863.319539:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773863.319542:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773863.319551:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664468096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773863.831199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773864.855146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:2.0:1775773865.111311:0:412974:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00000100:00080000:2.0:1775773865.111620:0:404331:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff982980f9ca80 x1862032664468224/t0(0) o103->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775773865.879178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773866.903131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773867.927177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773868.439166:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439177:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773868.439194:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439220:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439228:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773868.439269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664468352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439298:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773868.439312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439322:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439327:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773868.439354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664468480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773868.439373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439383:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773868.439404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664468608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773868.439424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439437:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773868.439456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664468736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439469:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773868.439483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439489:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439496:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439504:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439512:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773868.439529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664468864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439541:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439545:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773868.439548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439557:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439562:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439568:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439576:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773868.439591:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664468992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773868.439620:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773868.439629:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773868.439634:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773868.439642:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15635 00000100:00080000:1.0:1775773868.439647:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773868.439652:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773868.439662:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773868.439680:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664469120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773868.951154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773869.975216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773870.999194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773872.023180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773873.047154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773874.071151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773875.095142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773876.119216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773877.143279:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773878.167187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773878.679210:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679218:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773878.679223:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679232:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679250:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773878.679293:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664470144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679322:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679325:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773878.679326:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679330:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679333:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679336:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679344:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773878.679356:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664470272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773878.679371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679374:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679378:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679381:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773878.679388:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664470400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679395:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679397:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773878.679399:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679401:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679403:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679405:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773878.679413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664470528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773878.679422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679425:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679427:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679429:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679433:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773878.679448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664470656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679464:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679466:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773878.679467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679470:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679474:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679477:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773878.679485:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664470784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773878.679491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773878.679493:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773878.679494:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773878.679498:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15646 00000100:00080000:1.0:1775773878.679501:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773878.679503:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773878.679506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773878.679514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664470912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773879.191198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773880.215238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773881.239229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773882.263204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773883.287190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773884.311196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773885.335159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773886.359143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773887.383215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773888.407181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773889.431202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773890.455143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773891.479168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773892.503125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773893.527178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773894.551215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773895.575167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773896.599164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773897.623135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773898.647188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773899.671159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773900.695194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773901.719173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:1.0:1775773902.231353:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775773902.231391:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775773902.231414:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231460:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231469:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231476:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231483:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231492:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775773902.231504:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:2.0:1775773902.743139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773903.767163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773904.791196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773905.815170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773906.839217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773907.863163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773908.887247:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773909.911183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773910.935210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773911.959147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773912.983206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773914.007133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773915.031166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773916.055184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773917.079177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773918.103172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773919.127154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773920.151180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773921.175157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773922.199222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773923.223160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773924.247250:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773925.271150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773926.295262:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773927.319186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773928.343339:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773929.367215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773930.391192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773931.415178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773932.439179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773933.463176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773934.487158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775773934.999250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999260:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773934.999265:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999292:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999304:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999311:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773934.999353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664480000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999388:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773934.999393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999398:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999406:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773934.999425:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664480128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773934.999437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999440:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999443:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999448:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999455:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773934.999462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664480256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999474:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999476:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773934.999478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999480:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999482:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773934.999495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664480384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999504:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773934.999505:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999507:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999510:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773934.999529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664480512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773934.999539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999541:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999545:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999566:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999574:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773934.999588:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664480640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773934.999604:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773934.999630:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773934.999633:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773934.999638:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15702 00000100:00080000:2.0:1775773934.999643:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773934.999651:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773934.999660:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773934.999673:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664480768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773935.511249:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773936.535224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773937.559193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773938.583169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773939.607172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773940.631127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773941.655159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773942.679149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773943.703154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773944.727150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773945.751182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773946.775186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773947.799225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773948.823178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773949.847255:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773950.871189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773951.895176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773952.919165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773953.943211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773954.967267:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773955.991215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773957.015156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773958.039192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773959.063185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773960.087177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773961.111333:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773962.135159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773963.159225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773964.183223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773965.207181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773965.719156:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719167:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773965.719171:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719178:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719189:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719194:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773965.719226:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664485376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719259:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719263:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773965.719265:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719270:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719274:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719279:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719289:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773965.719304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664485504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773965.719324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719328:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719331:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719334:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773965.719348:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664485632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719354:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773965.719358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719362:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719373:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719375:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773965.719386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664485760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773965.719399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719404:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719407:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719412:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773965.719436:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664485888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773965.719461:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719468:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719472:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719476:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773965.719506:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664486016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773965.719514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773965.719519:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773965.719522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773965.719528:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15733 00000100:00080000:1.0:1775773965.719531:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773965.719534:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773965.719538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773965.719548:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664486144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775773966.231162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773967.255191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:1.0:1775773968.279173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:2.0:1775773968.791327:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775773968.791371:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775773968.791391:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791444:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791456:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791467:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791477:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791493:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775773968.791512:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775773969.303172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775773981.079260:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079277:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775773981.079286:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079302:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079324:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079336:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775773981.079400:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664488064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079443:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773981.079447:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079453:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079458:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079463:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773981.079491:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664488192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079503:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079507:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773981.079509:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079514:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079517:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079523:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079533:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773981.079554:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664488320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079573:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775773981.079575:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079579:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079582:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079588:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773981.079607:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664488448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079618:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079623:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775773981.079626:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079639:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079649:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079653:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079658:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775773981.079685:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664488576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079700:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079706:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775773981.079711:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079732:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079745:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079760:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079768:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773981.079786:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664488704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775773981.079849:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775773981.079860:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775773981.079872:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775773981.079881:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15748 00000100:00080000:1.0:1775773981.079887:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775773981.079894:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775773981.079902:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775773981.079918:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664488832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773983.895269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773986.199243:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199254:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773986.199258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199269:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199282:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199289:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773986.199341:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664488960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199371:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199375:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773986.199377:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199382:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199387:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199392:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773986.199420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664489088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773986.199442:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199447:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199451:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199455:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773986.199474:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664489216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199495:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773986.199497:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199502:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199507:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199512:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199517:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773986.199531:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664489344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199555:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199563:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773986.199567:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199574:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199580:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199588:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773986.199606:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664489472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199620:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199625:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773986.199627:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199632:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199636:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199641:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773986.199658:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664489600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773986.199672:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773986.199675:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773986.199676:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773986.199681:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15753 00000100:00080000:2.0:1775773986.199685:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773986.199692:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773986.199700:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773986.199711:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664489728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775773986.967251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773990.039234:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773993.111238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775773996.183196:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775773996.439186:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439207:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775773996.439212:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439221:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439233:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439239:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775773996.439279:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664490752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439307:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439311:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773996.439313:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439316:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439319:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773996.439359:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664490880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773996.439388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439401:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439418:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439425:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773996.439440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664491008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439453:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775773996.439457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439464:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439478:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439482:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773996.439499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664491136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439507:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439512:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775773996.439516:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439525:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439529:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439531:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775773996.439544:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664491264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439552:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439561:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775773996.439566:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439577:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439587:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439594:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439606:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773996.439627:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664491392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775773996.439644:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775773996.439648:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775773996.439650:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775773996.439655:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15763 00000100:00080000:2.0:1775773996.439661:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775773996.439665:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775773996.439674:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775773996.439690:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664491520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775773996.951185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773997.975207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775773998.999205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775773999.255313:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774000.023201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774001.047163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774002.071174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774002.327242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774003.095143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774004.119187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774005.143143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774005.399252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774006.167175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:1.0:1775774006.679210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679218:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774006.679222:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679231:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679248:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774006.679302:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664492544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679327:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774006.679329:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679332:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679335:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679338:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679346:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774006.679354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664492672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679361:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774006.679365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679368:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679370:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679372:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679375:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774006.679387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664492800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679402:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679406:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774006.679407:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679415:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774006.679439:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664492928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679458:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774006.679467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679479:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774006.679504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664493056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774006.679543:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679552:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679558:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679565:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679571:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774006.679584:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664493184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774006.679601:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774006.679605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774006.679607:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774006.679614:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15774 00000100:00080000:1.0:1775774006.679621:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774006.679628:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774006.679634:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774006.679649:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664493312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774007.191149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774008.215166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774008.471247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774009.239147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774010.263229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774011.287189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774011.543197:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774011.799148:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799157:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774011.799161:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799169:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799179:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799184:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774011.799219:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664493440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799241:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774011.799245:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799248:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799250:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799252:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799268:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774011.799277:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664493568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799289:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774011.799294:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799299:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799305:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799307:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799310:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774011.799318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664493696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799326:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774011.799327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799330:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799332:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799334:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774011.799344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664493824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799349:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799352:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774011.799354:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799356:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799359:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799361:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774011.799375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664493952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774011.799386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799388:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799393:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774011.799403:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664494080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774011.799411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774011.799418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774011.799421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774011.799429:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15779 00000100:00080000:1.0:1775774011.799434:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774011.799437:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774011.799440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774011.799448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664494208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774012.311249:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774013.335193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774014.359183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774015.383221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774016.407206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774017.431204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774018.455197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774019.479233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774020.503165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774021.527180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774022.551143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774023.575185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774024.599141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774025.623166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774026.647135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774027.671204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774028.695133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774029.719130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774030.743125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000400:00000100:1.0:1775774035.352662:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774035.352706:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774035.352724:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352765:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352774:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352785:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352797:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352809:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774035.352821:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00080000:2.0:1775774042.519223:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774042.519236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519245:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519257:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519264:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774042.519306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664498816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774042.519340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774042.519368:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664498944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774042.519380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519385:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774042.519407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664499072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519417:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774042.519418:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519421:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519451:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774042.519463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664499200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774042.519475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519478:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774042.519497:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664499328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519505:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774042.519516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519523:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519528:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519531:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519534:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774042.519552:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664499456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774042.519572:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774042.519576:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774042.519578:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774042.519584:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15810 00000100:00080000:2.0:1775774042.519591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774042.519597:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774042.519603:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774042.519618:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664499584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639144:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639155:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774047.639160:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639171:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639184:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639191:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774047.639229:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664499712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639263:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774047.639264:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639269:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639274:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639279:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639289:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774047.639306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664499840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774047.639324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639328:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639333:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639336:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774047.639354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664499968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639362:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774047.639371:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639375:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639379:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639383:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639389:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774047.639397:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664500096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639408:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774047.639409:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639415:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639420:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639425:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639431:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774047.639443:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664500224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639465:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774047.639470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639474:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639479:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639485:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774047.639497:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664500352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774047.639504:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774047.639507:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774047.639508:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774047.639511:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15815 00000100:00080000:2.0:1775774047.639514:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774047.639516:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774047.639522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774047.639532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664500480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774056.599270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774059.672234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774062.743303:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774062.999227:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999240:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774062.999246:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999259:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999275:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999283:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774062.999335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664502400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774062.999383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999399:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774062.999430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664502528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999453:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999455:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774062.999457:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999460:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999463:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999466:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999469:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774062.999479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664502656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999488:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774062.999489:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999492:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999497:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999499:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999503:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774062.999514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664502784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999525:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774062.999531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999535:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999540:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999546:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774062.999565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664502912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999580:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999589:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774062.999594:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999600:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999605:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999614:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999625:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774062.999647:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664503040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774062.999658:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774062.999663:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774062.999665:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774062.999673:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15830 00000100:00080000:2.0:1775774062.999679:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774062.999683:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774062.999689:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774062.999710:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664503168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774065.815297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774068.887295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774071.959247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774073.239199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239207:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774073.239211:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239230:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239236:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774073.239280:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664504192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239301:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774073.239305:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239311:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774073.239330:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664504320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239337:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774073.239341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239360:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774073.239369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664504448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774073.239386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239390:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239409:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239414:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774073.239437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664504576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239449:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774073.239455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239470:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239478:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774073.239490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664504704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239523:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774073.239533:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239540:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239547:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239554:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239561:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774073.239576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664504832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774073.239592:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774073.239596:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774073.239610:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774073.239616:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15840 00000100:00080000:2.0:1775774073.239623:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774073.239633:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774073.239642:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774073.239663:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664504960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774075.031256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774078.103281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774081.175219:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774082.967200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774083.991241:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774084.247311:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774085.015176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774086.039223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774087.063134:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774087.319257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774088.087195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00080000:2.0:1775774088.599220:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599233:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774088.599238:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599250:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599263:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599270:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774088.599307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664506880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774088.599370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599375:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599380:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599385:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774088.599410:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664507008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774088.599431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599442:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599446:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599451:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774088.599468:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664507136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599490:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774088.599492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599496:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599525:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599530:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599537:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774088.599556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664507264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599569:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599573:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774088.599575:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599581:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599586:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599591:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599597:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774088.599614:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664507392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599627:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599638:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774088.599642:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599657:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599664:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599674:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599685:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774088.599709:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664507520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774088.599720:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774088.599724:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774088.599726:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774088.599737:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15856 00000100:00080000:2.0:1775774088.599744:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774088.599749:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774088.599755:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774088.599770:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664507648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774089.111196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774090.135184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774090.391263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774091.159191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774093.463296:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774096.535290:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774099.607291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775774101.911445:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775774101.911494:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775774101.911517:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911583:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911594:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911604:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911613:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911622:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774101.911633:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00010000:00010000:1.0:1775774102.679273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774105.751295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774108.823210:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774111.895242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774114.967252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774118.039243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774119.319205:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319213:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774119.319217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319227:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319241:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319248:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774119.319291:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664512256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319312:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319316:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774119.319318:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319327:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774119.319357:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664512384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319372:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774119.319373:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319378:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319382:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319385:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319391:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774119.319403:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664512512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774119.319421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319430:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319435:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774119.319451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664512640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774119.319462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319470:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319477:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319482:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774119.319500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664512768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319520:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774119.319529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319534:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319538:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319541:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319546:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774119.319562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664512896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774119.319573:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774119.319576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774119.319580:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774119.319586:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15886 00000100:00080000:1.0:1775774119.319591:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774119.319594:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774119.319598:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774119.319610:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664513024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774121.111288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774124.183282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774124.439191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439200:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774124.439204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439215:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439228:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439234:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774124.439277:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664513152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439305:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774124.439306:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439309:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439311:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439314:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439322:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774124.439331:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664513280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439338:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439340:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774124.439341:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439347:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439348:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774124.439365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664513408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439374:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439377:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774124.439379:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439383:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439386:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439390:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439395:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774124.439408:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664513536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774124.439430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439433:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439437:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774124.439462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664513664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774124.439479:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439486:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439494:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439498:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439503:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774124.439514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664513792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774124.439530:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774124.439536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774124.439538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774124.439543:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15891 00000100:00080000:1.0:1775774124.439547:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774124.439551:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774124.439556:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774124.439575:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664513920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774124.951217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774125.975157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774126.999195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774127.255265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774128.023208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774129.047184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774130.071183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774130.327202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774131.095194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774132.119193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774133.399213:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774134.679165:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679177:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774134.679182:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679195:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679210:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774134.679269:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664514944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679305:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774134.679307:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679313:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679317:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679322:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679334:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774134.679349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664515072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679360:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774134.679361:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679364:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679367:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679369:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774134.679381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664515200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679387:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774134.679391:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679393:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679395:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679400:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774134.679417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664515328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679428:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774134.679434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679439:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679451:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679460:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774134.679474:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664515456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679496:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774134.679502:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679507:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679512:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679516:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679520:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774134.679543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664515584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774134.679551:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774134.679554:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774134.679555:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774134.679558:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15902 00000100:00080000:1.0:1775774134.679563:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774134.679570:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774134.679574:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774134.679584:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664515712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774136.471286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774139.543255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774139.799222:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799236:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774139.799246:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799259:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799274:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799282:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774139.799331:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664515840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799362:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799366:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774139.799368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799373:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799377:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799381:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799393:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774139.799408:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664515968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799428:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774139.799454:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799466:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774139.799501:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664516096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799513:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799517:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774139.799523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799531:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799537:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799543:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774139.799567:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664516224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799586:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799604:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774139.799607:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799618:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799629:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799635:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799642:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774139.799658:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664516352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799685:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799700:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774139.799706:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799720:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799729:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799738:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799749:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774139.799773:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664516480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774139.799811:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774139.799816:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774139.799819:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774139.799833:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15907 00000100:00080000:1.0:1775774139.799840:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774139.799847:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774139.799853:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774139.799865:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664516608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774142.615244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774144.919199:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919213:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774144.919218:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919232:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919254:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919265:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774144.919314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664516736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919356:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774144.919357:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919362:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919366:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919371:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774144.919395:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664516864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919409:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919413:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774144.919414:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919419:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919422:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919427:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774144.919444:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664516992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919457:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774144.919458:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919462:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919466:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919470:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774144.919487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664517120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919498:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919502:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774144.919504:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919508:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919513:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919516:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919521:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774144.919534:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664517248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919556:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919559:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774144.919561:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919564:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919567:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919572:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919578:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774144.919589:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664517376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774144.919602:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774144.919607:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774144.919609:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774144.919616:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15912 00000100:00080000:1.0:1775774144.919619:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774144.919623:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774144.919627:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774144.919635:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664517504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774145.431237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774145.687214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774146.455181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774147.479197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774148.503168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774148.759265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774149.527200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774150.551181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774151.575202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774151.831278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774152.599170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774153.623243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774154.647154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774154.903213:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774157.975270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774158.999251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774161.047244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774161.815224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774162.071253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774162.839152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774163.863206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774164.119273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774164.887132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774165.143259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774165.399197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399209:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774165.399214:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399240:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399249:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774165.399291:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664520320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399322:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399326:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774165.399328:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399333:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399338:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774165.399372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664520448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399386:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774165.399392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399397:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399406:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774165.399428:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664520576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399443:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774165.399445:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399449:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399455:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399460:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399466:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774165.399480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664520704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774165.399508:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399515:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399520:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399525:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399532:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774165.399552:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664520832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774165.399569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399576:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399585:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399591:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399598:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774165.399616:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664520960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774165.399639:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774165.399643:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774165.399646:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774165.399661:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15932 00000100:00080000:2.0:1775774165.399668:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774165.399677:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774165.399683:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774165.399698:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664521088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774165.911204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774166.935208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774167.191217:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774167.959193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774168.215338:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774168.471384:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774168.471429:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774168.471447:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471487:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471495:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471504:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471515:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471526:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774168.471539:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00010000:00010000:1.0:1775774170.263266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774171.287266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774173.335281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774174.359259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774175.639242:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639256:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774175.639263:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639276:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639292:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639301:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774175.639353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664522112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774175.639400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639410:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639416:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774175.639448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664522240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774175.639467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639477:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639482:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774175.639501:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664522368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639512:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639519:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774175.639522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639528:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639532:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639537:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639543:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774175.639554:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664522496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639571:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639580:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774175.639583:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639588:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639592:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639599:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639610:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774175.639627:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664522624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639639:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639643:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774175.639645:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639650:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639657:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639664:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639671:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774175.639690:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664522752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774175.639725:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774175.639736:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774175.639740:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774175.639751:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15943 00000100:00080000:2.0:1775774175.639783:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774175.639793:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774175.639805:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774175.639841:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664522880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774176.407256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774177.431263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774179.479250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774180.503208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774182.551229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774183.575268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774185.623236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774186.647275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774188.695242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774189.719216:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774190.999170:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999182:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774190.999187:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999200:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999218:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999227:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774190.999284:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664524800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999311:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999316:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774190.999318:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999328:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999344:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774190.999361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664524928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999374:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999378:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774190.999380:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999387:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999394:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774190.999417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664525056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999424:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774190.999428:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999431:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999434:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999436:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774190.999448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664525184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999457:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774190.999458:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999461:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999466:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999468:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999472:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774190.999483:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664525312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999492:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774190.999493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999498:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999503:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774190.999512:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664525440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774190.999523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774190.999526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774190.999527:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774190.999530:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15958 00000100:00080000:2.0:1775774190.999532:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774190.999535:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774190.999553:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774190.999562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664525568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774191.767276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774192.791260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774194.839257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774195.863249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774196.119178:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119189:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774196.119195:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119205:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119216:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119222:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774196.119263:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664525696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119294:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119297:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774196.119299:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119303:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119306:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119309:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119318:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774196.119330:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664525824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119338:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119342:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774196.119343:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119347:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119349:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119354:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774196.119366:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664525952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119378:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774196.119382:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119384:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119400:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119402:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774196.119419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664526080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119430:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774196.119431:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119435:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119438:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119441:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774196.119483:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664526208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119504:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774196.119505:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119511:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119516:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119521:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774196.119538:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664526336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774196.119544:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774196.119546:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774196.119548:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774196.119554:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15963 00000100:00080000:2.0:1775774196.119561:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774196.119565:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774196.119571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774196.119599:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664526464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774197.911248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774198.935298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774200.983270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774201.239235:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774201.239248:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239256:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239267:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239273:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774201.239309:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664526592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239335:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774201.239336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239339:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239341:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774201.239377:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664526720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774201.239392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239395:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239400:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239403:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774201.239418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664526848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774201.239434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239437:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239439:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774201.239465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664526976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774201.239477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239479:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239482:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774201.239499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664527104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774201.239514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239516:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239518:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239521:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774201.239530:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664527232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774201.239548:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774201.239551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774201.239553:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774201.239557:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15968 00000100:00080000:1.0:1775774201.239560:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774201.239563:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774201.239566:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774201.239573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664527360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774202.007225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774204.055254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774205.079265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774207.127304:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774208.151288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774210.199376:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774211.223305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774211.991246:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774213.015203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774213.271298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774214.039128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774214.295269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774216.343229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774217.367228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774219.415235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774220.439218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774221.719228:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719242:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774221.719248:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719259:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719273:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719282:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774221.719331:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664530176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719362:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774221.719372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719377:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719381:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719385:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719396:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774221.719409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664530304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719429:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719433:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774221.719436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719440:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719445:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719449:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719455:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774221.719470:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664530432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719481:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719485:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774221.719487:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719492:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719508:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774221.719530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664530560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719538:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719542:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774221.719546:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719555:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719563:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719570:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719576:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774221.719591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664530688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719605:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719611:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774221.719613:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719618:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719624:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719630:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719638:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774221.719649:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664530816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774221.719661:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774221.719664:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774221.719666:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774221.719671:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 15989 00000100:00080000:2.0:1775774221.719675:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774221.719679:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774221.719682:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774221.719699:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664530944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774222.487280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774223.511228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774225.559274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774226.583284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774228.631271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774229.655286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774231.703221:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774232.727288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774234.775261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775774235.031410:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775774235.031491:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775774235.031516:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031575:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031586:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031596:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031614:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031624:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774235.031633:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00010000:00010000:1.0:1775774235.799222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774237.079185:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079194:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774237.079198:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079207:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079223:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774237.079255:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664532864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079278:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079281:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774237.079282:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079286:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079288:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079291:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079299:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774237.079309:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664532992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774237.079325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079330:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079336:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079343:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079349:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774237.079362:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664533120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079386:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079392:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774237.079393:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079398:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079403:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079408:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079419:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774237.079433:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664533248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079444:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774237.079451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079459:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079465:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079473:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079482:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774237.079496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664533376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079532:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774237.079537:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079545:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079556:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079567:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774237.079589:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664533504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774237.079602:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774237.079607:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774237.079610:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774237.079617:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16004 00000100:00080000:1.0:1775774237.079624:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774237.079634:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774237.079643:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774237.079660:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664533632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774237.847279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774238.871233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774240.919262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774241.943261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774243.991276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774245.015222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774247.063277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774247.319196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319207:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774247.319211:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319222:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319237:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319244:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774247.319301:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664534656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319327:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319330:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774247.319331:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319335:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319338:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319340:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319351:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774247.319361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664534784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319369:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774247.319372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319376:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319378:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319380:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319383:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774247.319390:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664534912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319404:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774247.319408:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319413:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319415:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319417:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774247.319430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664535040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319437:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774247.319442:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319445:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319448:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319450:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319453:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774247.319460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664535168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319465:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319468:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774247.319469:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319472:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319480:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774247.319490:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664535296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774247.319497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774247.319500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774247.319501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774247.319504:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16014 00000100:00080000:1.0:1775774247.319507:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774247.319510:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774247.319513:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774247.319524:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664535424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774248.087260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774250.135262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774251.159318:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774253.207264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774254.231292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774256.279248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774257.303279:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774259.351285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774260.375228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774262.423223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774262.679196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679209:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774262.679214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679227:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679242:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679252:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774262.679336:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664537344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774262.679373:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679378:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679382:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679388:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774262.679423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664537472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774262.679442:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679446:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679451:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679456:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679462:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774262.679475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664537600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679486:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679493:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774262.679496:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679504:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679518:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679523:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679530:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774262.679547:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664537728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679572:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774262.679574:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679581:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679588:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679597:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679602:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774262.679620:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664537856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679632:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679636:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774262.679639:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679647:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679653:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679658:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679664:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774262.679683:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664537984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774262.679706:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774262.679712:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774262.679715:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774262.679725:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16030 00000100:00080000:2.0:1775774262.679732:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774262.679742:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774262.679751:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774262.679769:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664538112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774263.447250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774265.495298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774266.519276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774268.567269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774269.591270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774271.639332:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774272.663284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774274.711206:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774275.735246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774277.783200:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774278.807207:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774280.855249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774281.879198:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774283.159183:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159194:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774283.159200:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159210:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159224:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159231:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774283.159265:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664540928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159289:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774283.159290:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159293:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159296:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159298:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159306:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774283.159314:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664541056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774283.159323:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159326:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159327:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159332:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774283.159339:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664541184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159352:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159354:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774283.159356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159358:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159359:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159361:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774283.159371:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664541312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774283.159379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159382:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159385:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159387:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159389:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774283.159394:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664541440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159408:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774283.159409:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159415:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159419:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159421:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774283.159431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664541568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774283.159437:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774283.159439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774283.159440:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774283.159443:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16050 00000100:00080000:1.0:1775774283.159446:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774283.159448:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774283.159451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774283.159459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664541696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774283.671181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774283.927241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774284.695174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774284.951285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774285.719125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774286.743150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774286.999257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774287.767181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774288.023229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774288.279405:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16055 00000100:00080000:0.0:1775774288.279408:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774288.279410:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774288.279414:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774288.279421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664542208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774288.279426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774288.279429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774288.279430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774288.279432:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16055 00000100:00080000:0.0:1775774288.279434:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774288.279438:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774288.279443:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774288.279450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664542336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774288.279456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774288.279458:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774288.279460:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774288.279462:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16055 00000100:00080000:0.0:1775774288.279465:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774288.279467:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774288.279470:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774288.279486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664542464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774288.279517:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774288.279521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774288.279524:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774288.279527:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16055 00000100:00080000:0.0:1775774288.279531:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774288.279536:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774288.279539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774288.279548:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664542592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774288.791200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774289.047235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774289.815176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774290.071226:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774290.839150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774291.095222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774291.863173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774292.119259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774292.887144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774293.143307:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774293.399145:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399155:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774293.399160:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399169:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399181:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399187:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774293.399219:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664542720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399245:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399249:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774293.399251:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399255:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399259:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399263:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399272:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774293.399286:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664542848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399297:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399300:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774293.399302:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399306:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399310:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399313:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399317:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774293.399328:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664542976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399342:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774293.399347:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399351:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399355:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399358:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399363:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774293.399376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664543104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399410:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774293.399418:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399427:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399432:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399436:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399442:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774293.399454:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664543232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399464:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399468:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774293.399469:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399473:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399477:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399484:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399489:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774293.399500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664543360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774293.399520:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774293.399526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774293.399529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774293.399537:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16060 00000100:00080000:0.0:1775774293.399543:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774293.399548:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774293.399555:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774293.399571:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664543488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774293.911152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774294.167257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774294.935151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774295.191235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774295.959169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774296.215229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774296.983130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774297.239259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774298.007204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774298.263263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774298.519196:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774298.519213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519224:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519236:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774298.519280:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664543616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519309:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519313:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774298.519315:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519319:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519323:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519328:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774298.519350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664543744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774298.519378:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519387:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519390:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774298.519409:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664543872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519419:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774298.519426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519435:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519439:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774298.519457:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664544000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519466:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774298.519472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519477:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519484:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519490:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519496:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774298.519516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664544128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519544:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774298.519546:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519553:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519563:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519569:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519575:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774298.519591:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664544256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774298.519610:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774298.519613:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774298.519615:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774298.519619:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16066 00000100:00080000:1.0:1775774298.519623:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774298.519627:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774298.519632:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774298.519647:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664544384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774299.031204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774299.287203:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774300.055155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774300.311257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774301.079180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774301.335223:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774301.591667:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774301.591762:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774301.591797:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591861:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591873:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591884:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591897:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591908:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774301.591918:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:2.0:1775774302.103175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774302.359252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774303.127196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774303.383269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774303.639239:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639251:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774303.639255:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639267:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639281:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639302:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774303.639343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664544512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774303.639380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639395:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774303.639426:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664544640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639440:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774303.639446:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639452:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639456:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639460:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774303.639478:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664544768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639488:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639492:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774303.639494:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639498:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639502:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639507:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639513:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774303.639528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664544896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639541:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774303.639548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639552:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639558:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639562:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774303.639589:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664545024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639609:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639613:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774303.639615:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639620:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639625:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639631:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639641:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774303.639656:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664545152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774303.639667:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774303.639674:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774303.639677:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774303.639682:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16071 00000100:00080000:0.0:1775774303.639686:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774303.639690:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774303.639695:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774303.639708:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664545280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774304.151200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774304.407253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774305.175190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774305.431263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774306.199198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774306.455276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774307.223203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774307.481596:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774308.247226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774308.503250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774308.759419:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664545408:192.168.201.118@tcp:250:kworker.0 00000800:00000100:0.0:1775774308.759513:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759562:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9c000 x1862032664545408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664545408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759600:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759604:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664545408:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774308.759639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774308.759662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9f800 x1862032664545536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664545536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759688:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774308.759701:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774308.759720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9d180 x1862032664545664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759742:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664545664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759750:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774308.759776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545792:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774308.759796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9ce00 x1862032664545792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664545792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774308.759840:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774308.759860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9c700 x1862032664545920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759888:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664545920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759894:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759896:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759900:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664545920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774308.759905:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664546048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774308.759925:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.759935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9d880 x1862032664546048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.759947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664546048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.759954:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.759956:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.759958:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664546048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774308.759973:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664546176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774308.759991:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774308.760000:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774308/real 1775774308] req@ffff982980f9e300 x1862032664546176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774308.760011:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664546176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774308.760025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774308.760027:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774308.760030:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664546176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774309.271215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774309.527259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774310.295186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774310.551291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774311.319224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774311.575267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774312.343175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774312.599208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774313.367178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774313.623238:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774313.879206:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879214:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774313.879218:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879226:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879237:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774313.879275:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664546304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879296:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879299:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774313.879300:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879302:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879307:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879313:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774313.879320:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664546432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879326:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879330:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774313.879331:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879333:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879335:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879340:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774313.879351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664546560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774313.879360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879362:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879364:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879366:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774313.879377:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664546688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774313.879386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879388:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879390:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879392:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879395:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774313.879414:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664546816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879420:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879422:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774313.879423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879425:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879428:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879432:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879436:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774313.879444:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664546944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774313.879450:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774313.879452:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774313.879453:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774313.879455:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16081 00000100:00080000:0.0:1775774313.879458:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774313.879461:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774313.879463:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774313.879471:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664547072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774314.391170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774314.647255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774315.415202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774315.671230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774316.439178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774316.695193:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774317.463304:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774317.719249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774318.487202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774318.743205:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774318.999173:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999192:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774318.999197:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999208:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999220:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999226:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774318.999264:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664547200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999291:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999293:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774318.999295:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999298:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999302:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999318:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774318.999343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664547328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774318.999366:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999375:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999384:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774318.999396:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664547456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774318.999411:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999415:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999420:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999427:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999433:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774318.999446:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664547584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999463:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999470:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774318.999471:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999475:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999479:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999485:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999491:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774318.999501:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664547712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999514:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999518:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774318.999521:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999524:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999527:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999529:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999532:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774318.999538:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664547840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774318.999545:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774318.999548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774318.999550:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774318.999556:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16086 00000100:00080000:0.0:1775774318.999559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774318.999562:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774318.999565:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774318.999578:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664547968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774319.767264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774320.791205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774321.815253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774322.839287:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774323.863336:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774324.119226:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119238:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774324.119243:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119256:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119270:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119278:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774324.119327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664548096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119354:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119357:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774324.119359:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119363:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119366:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119381:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774324.119393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664548224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119405:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774324.119412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119417:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119420:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119423:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119428:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774324.119438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664548352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119446:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774324.119451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119455:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119459:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119463:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119468:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774324.119479:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664548480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119494:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774324.119496:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119501:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119504:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119508:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774324.119522:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664548608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119534:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119542:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774324.119545:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119550:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119554:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119560:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119566:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774324.119578:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664548736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774324.119599:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774324.119602:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774324.119603:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774324.119608:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16091 00000100:00080000:0.0:1775774324.119612:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774324.119617:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774324.119621:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774324.119634:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664548864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775774324.887256:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774325.911243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774326.935268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774327.959347:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774328.983247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774329.239243:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239254:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774329.239258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239270:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239282:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239289:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774329.239325:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664548992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239348:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774329.239353:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239358:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239363:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239367:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239376:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774329.239389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664549120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239400:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774329.239405:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239408:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239412:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239416:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239420:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774329.239430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664549248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239439:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239442:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774329.239444:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239453:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239457:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239461:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239467:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774329.239479:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664549376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239492:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774329.239502:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239507:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239512:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239517:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239525:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774329.239540:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664549504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239558:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239561:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774329.239563:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239567:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239571:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239576:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239583:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774329.239595:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664549632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774329.239605:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774329.239611:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774329.239614:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774329.239618:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16096 00000100:00080000:0.0:1775774329.239623:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774329.239628:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774329.239633:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774329.239647:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664549760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774330.007232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774331.031236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774332.055217:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774333.079279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775774334.103218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774334.359196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359208:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774334.359213:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359225:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359240:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359248:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774334.359300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664549888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359334:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774334.359340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359344:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359349:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774334.359381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664550016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774334.359402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359408:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359419:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359423:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359428:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774334.359443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664550144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359450:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359452:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774334.359454:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359457:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359460:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359463:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359468:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774334.359478:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664550272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359485:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359488:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774334.359489:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359493:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359496:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359499:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359502:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774334.359512:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664550400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359532:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359535:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774334.359537:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359543:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359551:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359556:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359565:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774334.359580:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664550528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774334.359593:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774334.359598:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774334.359600:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774334.359605:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16101 00000100:00080000:0.0:1775774334.359618:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774334.359635:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774334.359643:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774334.359658:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664550656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774335.127275:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774335.895230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774336.151204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774336.919235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774337.175265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774338.199204:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774339.223272:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774339.479203:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479214:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774339.479218:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479244:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479252:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774339.479297:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664550784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479329:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774339.479331:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479335:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479339:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479343:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479361:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774339.479375:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664550912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479391:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774339.479393:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479398:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479402:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479406:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479411:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774339.479423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664551040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479432:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479436:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774339.479437:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479441:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479445:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479453:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479460:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774339.479474:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664551168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479484:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774339.479489:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479495:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479509:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774339.479530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664551296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479546:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479551:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774339.479554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479561:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479568:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479574:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479579:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774339.479590:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664551424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774339.479607:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774339.479611:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774339.479613:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774339.479618:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16106 00000100:00080000:0.0:1775774339.479624:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774339.479631:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774339.479636:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774339.479651:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664551552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774340.247258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774341.271212:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774342.295243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774343.319252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774344.343220:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774344.599198:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599206:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774344.599210:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599220:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599232:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599238:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774344.599275:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664551680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599310:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599314:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774344.599316:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599324:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599327:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774344.599356:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664551808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599375:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774344.599376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599381:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599384:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599388:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774344.599406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664551936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774344.599421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599425:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599434:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774344.599452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664552064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774344.599466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599471:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599474:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599480:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774344.599496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664552192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599517:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774344.599523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599531:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599536:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599542:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599546:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774344.599554:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664552320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774344.599567:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774344.599571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774344.599573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774344.599577:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16112 00000100:00080000:0.0:1775774344.599581:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774344.599584:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774344.599589:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774344.599600:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664552448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774345.367225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774346.391287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774347.415252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774348.439220:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774349.463229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774349.719352:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664552576:192.168.201.118@tcp:250:kworker.0 00000800:00000100:0.0:1775774349.719427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719459:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9d500 x1862032664552576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719502:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664552576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719517:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719526:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719535:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664552576:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774349.719569:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774349.719597:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719609:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9c380 x1862032664552704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664552704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719623:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719625:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719627:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774349.719630:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774349.719640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9ea00 x1862032664552832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664552832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719661:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774349.719668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552960:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774349.719677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719682:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9e680 x1862032664552960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664552960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664552960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774349.719700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664553088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774349.719709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9df80 x1862032664553088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664553088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664553088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774349.719731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664553216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774349.719740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9f100 x1862032664553216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664553216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664553216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774349.719770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664553344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774349.719820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774349.719831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774349/real 1775774349] req@ffff982980f9fb80 x1862032664553344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774349.719838:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664553344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774349.719843:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774349.719844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774349.719846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664553344:192.168.201.118@tcp:38:kworker.0 00010000:00010000:0.0:1775774350.487230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774351.511214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774352.535249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774353.559188:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774354.583239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774354.839191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839197:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774354.839201:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839209:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839218:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839223:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774354.839252:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664553472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839273:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839276:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774354.839277:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839279:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839281:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839283:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774354.839298:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664553600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839308:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774354.839310:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839314:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839318:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839321:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839324:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774354.839333:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664553728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839338:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839340:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774354.839342:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839344:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839347:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839349:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839352:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774354.839363:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664553856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839380:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774354.839382:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839387:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839391:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839395:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839400:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774354.839414:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664553984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774354.839436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839442:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839447:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839455:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774354.839472:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664554112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774354.839490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774354.839493:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774354.839495:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774354.839499:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16122 00000100:00080000:0.0:1775774354.839505:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774354.839511:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774354.839515:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774354.839528:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664554240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774355.351177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774355.607197:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774356.375157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774356.631237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774357.399144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774357.655210:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774358.423229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774358.679289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774359.447251:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774359.703252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774359.959166:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959175:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774359.959180:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959191:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959204:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959211:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774359.959251:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664554368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959275:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959279:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774359.959281:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959295:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959300:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959304:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959313:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774359.959324:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664554496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774359.959337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959341:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959344:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959347:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959351:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774359.959367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664554624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959383:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774359.959388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959391:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959394:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959397:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959402:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774359.959413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664554752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959435:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774359.959436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959440:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959446:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774359.959459:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664554880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774359.959473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959476:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959479:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959483:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959486:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774359.959495:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664555008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774359.959503:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774359.959507:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774359.959508:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774359.959512:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16127 00000100:00080000:0.0:1775774359.959515:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774359.959518:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774359.959523:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774359.959532:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664555136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774360.471190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774360.727239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774361.495194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774361.751284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774362.519176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774362.775219:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774363.543203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774363.799332:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774364.567202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774364.823255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774365.079141:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079150:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774365.079154:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079164:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079177:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079184:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774365.079222:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664555264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079245:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079248:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774365.079249:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079253:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079255:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079257:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079267:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774365.079276:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664555392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079292:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774365.079296:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079301:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079309:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079313:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774365.079327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664555520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079336:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079339:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774365.079340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079343:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079347:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079350:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079355:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774365.079366:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664555648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774365.079383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079387:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079391:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079397:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079406:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774365.079421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664555776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079440:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079444:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774365.079446:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079452:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079459:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079464:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079471:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774365.079486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664555904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774365.079498:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774365.079514:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774365.079516:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774365.079524:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16132 00000100:00080000:0.0:1775774365.079529:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774365.079533:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774365.079538:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774365.079550:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664556032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774365.591204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774365.847270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774366.615168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774366.871264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774367.639180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774367.895287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775774368.151411:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775774368.151496:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775774368.151520:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151572:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151583:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151596:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151605:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151614:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774368.151622:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:2.0:1775774368.663157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774368.919255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774369.687242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774369.943241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774370.199201:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199213:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774370.199217:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199228:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199254:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774370.199294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664556160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774370.199324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199330:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199332:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199345:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774370.199358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664556288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774370.199375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199399:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774370.199411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664556416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774370.199438:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199445:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199449:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774370.199464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664556544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199476:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774370.199478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199482:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199486:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199489:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774370.199504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664556672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199513:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199516:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774370.199528:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199532:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199536:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199541:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199547:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774370.199561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664556800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774370.199573:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774370.199577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774370.199579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774370.199586:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16137 00000100:00080000:1.0:1775774370.199604:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774370.199612:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774370.199620:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774370.199638:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664556928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775774370.711200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774370.967243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774371.735202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774371.991269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774372.759148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00000100:00100000:2.0:1775774373.783180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774374.039383:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774374.807218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774375.063282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774375.319197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319209:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774375.319215:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319241:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319249:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774375.319294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664557056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774375.319342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319347:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319352:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319358:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774375.319385:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664557184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774375.319399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319402:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319405:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319408:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774375.319420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664557312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319429:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774375.319430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319432:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319435:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319438:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774375.319455:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664557440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319464:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774375.319465:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319469:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319471:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319473:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774375.319484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664557568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319508:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774375.319513:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319517:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319523:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319532:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774375.319553:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664557696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774375.319579:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774375.319583:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774375.319585:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774375.319592:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16142 00000100:00080000:0.0:1775774375.319597:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774375.319601:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774375.319607:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774375.319626:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664557824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774377.111217:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774378.135260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774380.183254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774380.439226:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439250:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774380.439254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439265:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439278:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439283:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774380.439325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664557952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439351:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439354:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774380.439355:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439361:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439368:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774380.439393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664558080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439407:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439410:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774380.439411:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439417:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439420:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439423:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439427:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774380.439440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664558208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774380.439455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439460:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439472:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774380.439488:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664558336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439497:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774380.439501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439505:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439508:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439510:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439514:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774380.439522:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664558464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439539:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774380.439548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439555:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439557:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439560:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439564:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774380.439573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664558592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774380.439582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774380.439588:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774380.439591:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774380.439598:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16147 00000100:00080000:0.0:1775774380.439601:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774380.439603:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774380.439606:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774380.439614:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664558720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774381.207263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774383.255270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774384.279262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774385.559258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559271:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774385.559276:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559289:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559312:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559319:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774385.559366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664558848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774385.559402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559410:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559414:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774385.559440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664558976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774385.559460:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559464:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559468:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559474:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559479:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774385.559493:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664559104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774385.559508:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559512:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559516:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559522:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774385.559542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664559232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559552:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559555:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774385.559557:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559561:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559565:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559570:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559575:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774385.559588:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664559360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559599:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559604:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774385.559608:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559617:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559623:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559630:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559638:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774385.559654:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664559488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774385.559664:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774385.559670:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774385.559673:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774385.559679:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16153 00000100:00080000:1.0:1775774385.559684:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774385.559687:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774385.559692:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774385.559709:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664559616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775774386.327226:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774387.351284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774389.399221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1775774390.423184:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774390.679187:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679194:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774390.679198:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679217:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679221:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774390.679252:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664559744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679272:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679276:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774390.679277:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679283:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679287:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679291:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679307:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774390.679318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664559872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679327:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679329:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774390.679331:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679334:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679337:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679340:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774390.679351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664560000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679360:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774390.679362:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679365:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679368:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679371:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679375:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774390.679383:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664560128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774390.679399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679402:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679408:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774390.679420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664560256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774390.679432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679438:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679441:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679445:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774390.679452:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664560384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774390.679459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774390.679462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774390.679463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774390.679466:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16158 00000100:00080000:2.0:1775774390.679469:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774390.679472:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774390.679476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774390.679484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664560512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774390.679602:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664560128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774390.679610:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774390.679612:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774390.679614:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774390.679616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774390.679625:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774390.679629:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774390/real 1775774390] req@ffff9829764b6d80 x1862032664560256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774390.679635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664560256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774390.679640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774390.679641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774390.679643:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774390.679646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664560384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774390.679654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774390.679659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774390/real 1775774390] req@ffff9829764b6a00 x1862032664560384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774390.679668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664560384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774390.679675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774390.679677:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774390.679681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664560384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774390.679699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664560512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774390.679717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774390.679725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774390/real 1775774390] req@ffff9829764b5c00 x1862032664560512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774390.679732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664560512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774390.679737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774390.679739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774390.679741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664560512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774391.191189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774391.447217:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774392.215138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774392.471204:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774393.239186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774393.495284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774394.263126:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774394.519232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774395.287204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774395.543306:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774395.799235:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799248:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774395.799255:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774395.799270:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799286:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799295:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774395.799347:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664560640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774395.799382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774395.799389:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774395.799394:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799399:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799405:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774395.799417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774395.799434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664560768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774395.799448:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774395.799460:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774395.799463:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664560640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774395.799468:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799479:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799485:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774395.799491:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774395.799506:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664560896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774395.799514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799517:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774395.799518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774395.799521:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799533:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799535:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774395.799540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774395.799545:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774395.799563:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664561024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774395.799576:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799581:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774395.799584:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774395.799585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b4700 x1862032664560640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774395.799600:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799612:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799618:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774395.799624:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664560640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774395.799626:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774395.799646:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664561152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774395.799654:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.799661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.799677:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664560640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774395.799687:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799691:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774395.799693:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774395.799698:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799701:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799704:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774395.799709:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774395.799714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774395.799728:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664561280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774395.799737:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774395.799742:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774395.799744:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774395.799750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774395.799754:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16163 00000100:00080000:2.0:1775774395.799761:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774395.799767:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774395.799772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b6680 x1862032664560768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774395.799774:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774395.799796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664560768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774395.799804:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664561408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774395.799807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.799817:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.799828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774395.799844:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774395.799873:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774395.799882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b5500 x1862032664560896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774395.799891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664560896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774395.799899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.799903:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.799906:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664560896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774395.799912:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774395.799933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774395.799943:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b7100 x1862032664561024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774395.799953:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664561024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774395.799961:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.799964:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.799982:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774395.799988:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774395.800008:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774395.800018:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b7b80 x1862032664561152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774395.800029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664561152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774395.800037:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.800040:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.800043:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774395.800103:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664561280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774395.800121:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774395.800129:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b4e00 x1862032664561280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774395.800136:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664561280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774395.800141:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.800144:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.800147:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664561280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774395.800152:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664561408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774395.800169:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774395.800179:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774395/real 1775774395] req@ffff9829764b4000 x1862032664561408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774395.800189:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664561408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774395.800197:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774395.800200:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774395.800203:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664561408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774396.311211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774396.567291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774397.335185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774397.591270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774398.359176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774398.615295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774399.383163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774399.639290:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774400.407196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774400.663289:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774400.919164:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919173:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774400.919178:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919189:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919202:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919208:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774400.919260:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664561536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774400.919291:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774400.919297:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919300:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919308:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774400.919318:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774400.919330:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664561664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774400.919339:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774400.919341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664561536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774400.919343:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774400.919345:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919352:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919354:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919357:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774400.919359:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774400.919367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664561792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774400.919374:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919376:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774400.919378:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919381:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919384:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919386:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774400.919391:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774400.919401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664561920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774400.919410:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774400.919416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919425:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919429:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774400.919434:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774400.919442:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774400.919449:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664562048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774400.919461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774400.919467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774400.919470:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919475:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774400.919480:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774400.919487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774400.919487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2e680 x1862032664561536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774400.919502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664562176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774400.919514:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664561536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774400.919525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774400.919532:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774400.919535:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774400.919536:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774400.919540:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16168 00000100:00080000:0.0:1775774400.919546:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774400.919546:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774400.919553:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664561536:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775774400.919563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774400.919580:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664562304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774400.919592:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774400.919624:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2e300 x1862032664561664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.919650:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664561664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.919659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.919662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.919666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774400.919673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774400.919696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2f480 x1862032664561792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.919730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664561792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.919741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.919744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.919748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774400.919756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774400.919779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2c380 x1862032664561920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.919805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664561920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.919814:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.919818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.919822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664561920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774400.919828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774400.919849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919860:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2ca80 x1862032664562048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.919873:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664562048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.919916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.919920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.919924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774400.919929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664562176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774400.919947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.919955:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2f100 x1862032664562176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.919963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664562176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.919969:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.919972:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.919975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664562176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774400.919997:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664562304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774400.920019:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774400.920031:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774400/real 1775774400] req@ffff982974b2fb80 x1862032664562304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774400.920043:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664562304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774400.920051:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774400.920053:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774400.920057:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664562304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774401.431196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774401.687280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774402.455197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774402.711297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774403.479190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774403.735273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774404.503213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774404.759216:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774405.527153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774405.783279:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774406.039226:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039238:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774406.039245:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039258:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039273:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039282:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774406.039334:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664562432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774406.039382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039387:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774406.039389:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039399:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039405:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039420:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774406.039436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664562560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774406.039449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039455:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774406.039457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774406.039457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664562432:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774406.039465:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039470:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039474:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039479:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774406.039502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664562688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774406.039513:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039518:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774406.039520:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039524:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039528:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039533:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774406.039552:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774406.039554:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664562816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774406.039566:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774406.039573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039579:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039582:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039587:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039594:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774406.039604:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2d180 x1862032664562432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774406.039608:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664562944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774406.039638:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039642:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774406.039644:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039650:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00100000:3.0:1775774406.039650:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664562432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774406.039657:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039664:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039669:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774406.039683:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664563072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774406.039684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.039695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.039703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664562432:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774406.039711:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774406.039715:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774406.039718:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774406.039723:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16173 00000100:00080000:0.0:1775774406.039727:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774406.039731:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774406.039738:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774406.039740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774406.039753:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664563200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774406.039775:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.039806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2df80 x1862032664562560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.039822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664562560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.039831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.039836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.039839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774406.039846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774406.039869:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.039881:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2d500 x1862032664562688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.039895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664562688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.039905:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.039918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.039923:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774406.039930:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774406.039955:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.039967:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2fb80 x1862032664562816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.039979:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664562816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.039988:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.040002:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.040006:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774406.040012:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774406.040033:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.040045:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2f100 x1862032664562944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.040057:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664562944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.040091:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.040095:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.040099:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664562944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774406.040105:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774406.040126:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.040138:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2ca80 x1862032664563072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.040150:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664563072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.040158:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.040162:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.040166:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774406.040188:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774406.040210:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774406.040222:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774406/real 1775774406] req@ffff982974b2c380 x1862032664563200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774406.040242:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664563200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774406.040251:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774406.040254:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774406.040258:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774406.551201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774406.807286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774407.575180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774407.831277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774408.599202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774408.855300:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774409.623161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774409.879238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774410.647228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774410.903242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774411.159227:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159237:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774411.159242:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159254:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159267:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159274:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774411.159328:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664563328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774411.159360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159363:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774411.159365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159369:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159373:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159377:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159388:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774411.159400:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664563456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774411.159410:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774411.159412:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664563328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774411.159413:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774411.159417:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159424:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159427:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159429:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774411.159441:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664563584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774411.159453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774411.159457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159460:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159481:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159484:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774411.159498:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664563712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774411.159499:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774411.159509:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159512:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774411.159514:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159517:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159519:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159522:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159525:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774411.159532:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664563840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774411.159539:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159542:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774411.159543:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774411.159546:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159551:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774411.159551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b4000 x1862032664563328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774411.159557:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774411.159571:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664563968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774411.159579:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774411.159582:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774411.159584:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774411.159587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664563328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774411.159590:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16178 00000100:00080000:2.0:1775774411.159594:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774411.159601:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774411.159606:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774411.159613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775774411.159617:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664564096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774411.159623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.159632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664563328:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774411.159678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774411.159743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.159759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b4e00 x1862032664563456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.159774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664563456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.159784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.159788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.159792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774411.159799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774411.159824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.159836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b7b80 x1862032664563584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.159850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664563584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.159860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.159863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.159867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774411.159874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774411.159898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.159910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b7100 x1862032664563712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.159932:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664563712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.159942:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.159946:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.159949:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774411.159956:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774411.159979:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.159990:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b5500 x1862032664563840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.160003:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664563840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.160009:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.160011:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.160013:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664563840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774411.160018:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774411.160032:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.160038:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b6680 x1862032664563968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.160046:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664563968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.160055:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.160058:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.160060:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664563968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774411.160063:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774411.160097:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774411.160104:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774411/real 1775774411] req@ffff9829764b4700 x1862032664564096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774411.160111:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664564096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774411.160116:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774411.160118:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774411.160120:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774411.671225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774411.927246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774412.695186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774412.951279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774413.719193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774413.975235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774414.743176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774414.999301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774415.767176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774416.023252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774416.279201:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774416.279209:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774416.279214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774416.279224:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16183 00000100:00080000:3.0:1775774416.279238:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279246:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774416.279286:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664564224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774416.279319:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774416.279323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774416.279325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774416.279328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16183 00000100:00080000:3.0:1775774416.279332:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774416.279343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774416.279354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664564352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774416.279365:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774416.279369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774416.279372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774416.279373:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664564224:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774416.279381:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16183 00000100:00080000:3.0:1775774416.279385:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279389:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774416.279395:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774416.279409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664564480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774416.279420:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774416.279424:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774416.279427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774416.279434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16183 00000800:00000100:0.0:1775774416.279436:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774416.279443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279449:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774416.279456:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774416.279468:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9ed80 x1862032664564224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774416.279473:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664564608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774416.279490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774416.279492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664564224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774416.279499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774416.279504:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774416.279511:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774416.279512:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16183 00000100:00080000:0.0:1775774416.279518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774416.279520:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774416.279522:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664564224:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774416.279529:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774416.279535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774416.279545:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774416.279550:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664564736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774416.279563:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774416.279566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774416.279571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774416.279573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774416.279578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16183 00000100:00100000:0.0:1775774416.279580:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9f480 x1862032664564352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774416.279585:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279594:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279596:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664564352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774416.279604:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774416.279639:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775774416.279642:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664564864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774416.279644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774416.279660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774416.279667:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774416.279670:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774416.279673:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774416.279678:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16183 00000800:00000100:0.0:1775774416.279682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774416.279687:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774416.279693:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9dc00 x1862032664564480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774416.279701:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774416.279716:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664564992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774416.279719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664564480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774416.279745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774416.279749:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774416.279758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774416.279773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9fb80 x1862032664564608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774416.279787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664564608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774416.279793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774416.279795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774416.279809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774416.279819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9f100 x1862032664564736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774416.279832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664564736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774416.279837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774416.279839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664564736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774416.279847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774416.279865:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9df80 x1862032664564864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774416.279888:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664564864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774416.279897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774416.279900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774416.279910:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774416.279929:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774416.279937:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774416/real 1775774416] req@ffff982980f9e680 x1862032664564992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774416.279945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664564992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774416.279950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774416.279952:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774416.279955:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664564992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774416.791139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774417.047289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774417.815211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774418.071324:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774418.839216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774419.095273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774419.863218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774420.119277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774420.887181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774421.143312:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774421.399176:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399186:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774421.399191:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399202:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399217:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399224:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774421.399276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664565120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774421.399301:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399305:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774421.399307:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399310:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399313:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399316:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399325:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774421.399337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664565248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774421.399345:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399348:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774421.399349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399355:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774421.399358:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664565120:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774421.399359:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774421.399373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664565376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774421.399379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774421.399388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399391:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399393:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399396:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774421.399411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664565504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774421.399418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774421.399422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399424:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399429:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399432:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774421.399441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664565632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774421.399449:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774421.399451:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774421.399455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774421.399459:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399462:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399468:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774421.399475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664565760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774421.399483:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774421.399486:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774421.399488:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774421.399489:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9ea00 x1862032664565120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774421.399495:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16188 00000100:00080000:3.0:1775774421.399501:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774421.399506:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774421.399510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774421.399516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664565888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774421.399538:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664565120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664565120:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774421.399596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774421.399616:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9c380 x1862032664565248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399636:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664565248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774421.399650:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774421.399661:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9d500 x1862032664565376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399675:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664565376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399680:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399683:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399685:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774421.399688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774421.399701:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9e300 x1862032664565504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664565504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774421.399742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774421.399765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9d880 x1862032664565632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664565632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399790:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399792:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664565632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774421.399798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664565760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774421.399813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9c700 x1862032664565760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664565760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664565760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774421.399839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664565888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774421.399850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774421.399855:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774421/real 1775774421] req@ffff982980f9ce00 x1862032664565888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774421.399862:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664565888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774421.399867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774421.399869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774421.399871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664565888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774421.911231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774422.167291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774422.935227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774423.191286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774423.959216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774424.215279:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774424.983164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774425.239268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774426.007196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774426.263251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774426.519109:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519117:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774426.519121:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519130:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:3.0:1775774426.519141:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519148:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774426.519181:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664566016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774426.519210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519214:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774426.519216:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519220:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:3.0:1775774426.519224:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519228:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774426.519237:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774426.519249:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664566144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774426.519255:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664566016:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774426.519259:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519261:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774426.519263:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519266:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:3.0:1775774426.519269:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519272:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774426.519276:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774426.519283:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664566272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774426.519291:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774426.519294:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519298:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:3.0:1775774426.519303:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519306:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774426.519310:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774426.519312:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774426.519319:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664566400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774426.519326:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519330:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774426.519331:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519334:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:3.0:1775774426.519337:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519340:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519342:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9d180 x1862032664566016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774426.519343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774426.519351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664566528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774426.519357:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664566016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774426.519358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774426.519363:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519365:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00080000:0.0:1775774426.519368:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774426.519368:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519371:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774426.519372:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775774426.519375:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774426.519376:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664566016:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774426.519381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664566656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774426.519387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774426.519390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774426.519391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774426.519394:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16194 00000100:00100000:0.0:1775774426.519396:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566144:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774426.519397:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774426.519400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774426.519404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774426.519412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664566784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774426.519415:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9f800 x1862032664566144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519458:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664566144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519463:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519465:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519467:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774426.519471:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774426.519486:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9c000 x1862032664566272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519499:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664566272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519505:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519507:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774426.519510:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774426.519519:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519525:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9ce00 x1862032664566400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519531:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664566400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519536:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519539:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774426.519542:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774426.519551:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9c700 x1862032664566528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519562:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664566528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519566:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664566528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774426.519572:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664566656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774426.519579:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519584:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9d880 x1862032664566656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664566656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664566656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774426.519608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664566784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774426.519617:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774426.519622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774426/real 1775774426] req@ffff982980f9e300 x1862032664566784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774426.519627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664566784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774426.519632:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774426.519634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774426.519635:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664566784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774427.031139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774427.287267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774428.055190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774428.311246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774429.079150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774429.335248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774430.103211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774430.359274:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774431.127202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774431.383228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774431.639139:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639148:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774431.639153:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639163:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639175:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639181:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774431.639229:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664566912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774431.639253:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639257:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774431.639258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639262:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639265:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639270:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639280:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774431.639301:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664567040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774431.639309:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664566912:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774431.639320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774431.639325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639329:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639332:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639339:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774431.639361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664567168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774431.639369:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639372:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774431.639374:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639377:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639380:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639384:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639389:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774431.639398:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664567296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774431.639407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774431.639411:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639415:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639418:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639421:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639425:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774431.639435:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664567424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774431.639445:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567040:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774431.639445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639449:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774431.639450:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639454:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00080000:3.0:1775774431.639457:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639460:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639464:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774431.639474:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774431.639475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664567552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774431.639484:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774431.639487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774431.639489:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774431.639492:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16199 00000100:00100000:0.0:1775774431.639493:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567296:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775774431.639496:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774431.639508:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774431.639513:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774431.639519:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774431.639523:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664567680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774431.639548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664567552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774431.639574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664567680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774432.151282:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774432.407298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774433.175198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774433.431184:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774434.199177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774434.455241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774434.711347:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774434.711391:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774434.711408:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711445:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711452:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711456:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711461:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711467:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774434.711471:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774434.711508:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9fb80 x1862032664567680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711534:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664567680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711550:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711554:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664567680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774434.711560:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9df80 x1862032664567424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711571:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664567424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711576:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774434.711584:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9f100 x1862032664567552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711595:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664567552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664567552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774434.711614:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9c380 x1862032664567040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711624:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664567040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711631:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711637:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774434.711642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9ea00 x1862032664567168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664567168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774434.711671:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9e680 x1862032664567296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664567296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711688:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774434.711698:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774431/real 1775774434] req@ffff982980f9d500 x1862032664566912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774434.711709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664566912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774434.711715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774434.711718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774434.711721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664566912:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774435.223202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774435.479237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774436.247193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774436.503237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774436.759220:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759229:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774436.759233:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759240:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759250:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759267:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774436.759299:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664567808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774436.759334:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759339:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774436.759340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759345:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759349:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759362:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774436.759374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664567936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774436.759384:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759387:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774436.759388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759396:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759399:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759404:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774436.759407:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664567808:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774436.759417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664568064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774436.759426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759430:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774436.759432:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759435:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774436.759461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664568192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774436.759470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759473:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774436.759475:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759478:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759482:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759485:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759489:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774436.759498:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774436.759500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664568320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774436.759509:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759512:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774436.759514:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759518:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759521:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759525:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774436.759538:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664568448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774436.759550:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2f480 x1862032664567808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774436.759552:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774436.759559:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774436.759562:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774436.759571:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16204 00000100:00080000:0.0:1775774436.759576:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774436.759580:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774436.759584:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774436.759590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664567808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774436.759597:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664568576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774436.759612:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.759621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.759627:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664567808:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774436.759637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774436.759672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.759690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2e300 x1862032664567936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.759707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664567936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.759715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.759719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.759723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664567936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774436.759754:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774436.759787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.759804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2e680 x1862032664568064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.759819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664568064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.759828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.759831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.759835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774436.759842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774436.759983:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.760001:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2c380 x1862032664568192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.760019:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664568192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.760028:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.760032:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.760036:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774436.760043:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774436.760110:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.760128:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2ca80 x1862032664568320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.760144:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664568320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.760152:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.760156:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.760159:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774436.760166:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664568448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774436.760195:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.760207:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2f100 x1862032664568448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.760221:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664568448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.760231:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.760234:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.760237:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664568448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774436.760245:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664568576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774436.760280:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774436.760292:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774436/real 1775774436] req@ffff982974b2fb80 x1862032664568576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774436.760302:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664568576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774436.760310:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774436.760312:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774436.760315:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664568576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774437.271165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774437.527225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774438.295146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774438.551241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774439.319229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774439.575280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774440.343163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774440.599228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774441.367186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774441.623258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774441.879208:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774441.879221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879231:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879244:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774441.879251:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774441.879298:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664568704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774441.879333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879337:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774441.879340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879344:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879349:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774441.879353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774441.879364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774441.879380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664568704:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774441.879381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664568832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774441.879396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774441.879401:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879405:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879409:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774441.879413:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774441.879418:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774441.879432:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664568960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774441.879442:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879446:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774441.879447:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879452:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879456:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774441.879460:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774441.879466:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774441.879466:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774441.879484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664569088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774441.879495:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774441.879501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774441.879502:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2d500 x1862032664568704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774441.879510:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879515:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774441.879519:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664568704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774441.879523:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774441.879533:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879538:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774441.879539:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664569216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774441.879541:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664568704:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774441.879554:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774441.879558:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774441.879560:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879565:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:0.0:1775774441.879569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774441.879569:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568832:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774441.879577:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774441.879582:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774441.879588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774441.879596:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664569344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774441.879600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2df80 x1862032664568832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774441.879614:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774441.879618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664568832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774441.879619:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774441.879622:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774441.879630:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16209 00000100:00080000:3.0:1775774441.879631:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774441.879633:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774441.879636:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774441.879639:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879642:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568832:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775774441.879644:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774441.879648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774441.879654:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664569472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774441.879662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2d180 x1862032664568960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774441.879677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664568960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774441.879683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774441.879687:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664568960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774441.879691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774441.879702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2fb80 x1862032664569088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774441.879712:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664569088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774441.879716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774441.879720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774441.879723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774441.879732:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2f100 x1862032664569216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774441.879742:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664569216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774441.879746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774441.879749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774441.879759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664569344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774441.879768:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2ca80 x1862032664569344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774441.879779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664569344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774441.879783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774441.879787:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664569344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774441.879789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664569472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774441.879797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774441.879802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774441/real 1775774441] req@ffff982974b2c380 x1862032664569472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774441.879808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664569472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774441.879813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774441.879814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774441.879816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664569472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774442.391178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774442.647243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774443.415191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774443.671270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774444.439143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774444.695261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774445.463229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774445.719300:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774446.487158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774446.743221:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774446.999228:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999239:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774446.999245:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774446.999257:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999272:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999280:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774446.999332:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664569600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774446.999365:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999368:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774446.999369:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774446.999373:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999380:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774446.999391:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774446.999404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664569728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774446.999417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999420:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774446.999423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774446.999428:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664569600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774446.999430:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999434:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999440:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774446.999449:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774446.999465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664569856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774446.999475:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999479:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774446.999480:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774446.999483:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999487:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774446.999494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774446.999506:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664569984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774446.999511:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774446.999522:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774446.999530:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774446.999536:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999542:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999546:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2e680 x1862032664569600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774446.999554:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774446.999574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664569600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774446.999580:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664570112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774446.999586:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774446.999591:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774446.999594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774446.999595:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774446.999598:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774446.999601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664569600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774446.999605:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999611:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999615:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774446.999620:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774446.999628:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664570240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774446.999633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569728:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774446.999644:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774446.999647:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774446.999657:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774446.999657:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774446.999665:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16214 00000100:00080000:0.0:1775774446.999669:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774446.999672:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774446.999677:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774446.999682:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2e300 x1862032664569728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774446.999689:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664570368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774446.999699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664569728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774446.999716:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774446.999730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2f480 x1862032664569856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774446.999746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664569856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999755:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774446.999759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774446.999772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2c380 x1862032664569984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774446.999791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664569984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999803:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664569984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774446.999813:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774446.999834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999844:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2ca80 x1862032664570112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774446.999852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664570112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774446.999874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664570240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774446.999887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999893:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2f100 x1862032664570240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774446.999900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664570240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999905:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999911:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664570240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774446.999916:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664570368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774446.999934:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774446.999945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774446/real 1775774446] req@ffff982974b2fb80 x1862032664570368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774446.999958:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664570368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774446.999967:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774446.999970:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774446.999974:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664570368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774447.511167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774447.767287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774448.535198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774448.791198:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774449.559200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774449.815284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774450.583230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774450.839247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774451.607165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774451.863231:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774452.119187:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119198:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774452.119203:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:3.0:1775774452.119223:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119228:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774452.119261:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664570496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119292:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774452.119294:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119298:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:3.0:1775774452.119302:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119306:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774452.119315:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774452.119327:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664570496:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774452.119327:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664570624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119347:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774452.119349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119353:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:3.0:1775774452.119358:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119361:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774452.119367:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774452.119378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664570752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774452.119391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119394:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:3.0:1775774452.119397:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774452.119398:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774452.119402:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774452.119407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774452.119418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664570880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774452.119428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119431:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:3.0:1775774452.119434:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119437:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119438:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9dc00 x1862032664570496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774452.119440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774452.119451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664571008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774452.119462:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664570496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774452.119470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774452.119474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774452.119477:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774452.119481:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16219 00000100:00080000:0.0:1775774452.119483:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119488:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664570496:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774452.119488:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119497:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774452.119502:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774452.119511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774452.119515:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664571136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774452.119523:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774452.119526:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774452.119528:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774452.119531:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16219 00000800:00000100:0.0:1775774452.119534:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774452.119535:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774452.119539:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774452.119544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774452.119546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9f480 x1862032664570624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774452.119555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664571264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774452.119556:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664570624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119581:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119588:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119591:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774452.119598:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774452.119619:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119630:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9ed80 x1862032664570752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774452.119641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664570752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119649:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774452.119661:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774452.119681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119691:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9d500 x1862032664570880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774452.119702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664570880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119709:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664570880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774452.119721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774452.119737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9e680 x1862032664571008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774452.119751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664571008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774452.119772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664571136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774452.119800:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9ea00 x1862032664571136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774452.119821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664571136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664571136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774452.119839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664571264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774452.119857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774452.119865:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774452/real 1775774452] req@ffff982980f9c380 x1862032664571264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774452.119880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664571264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774452.119885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774452.119886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774452.119888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664571264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774452.631168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774452.887200:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774453.655171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774453.911230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774454.679168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774454.935263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774455.703162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774455.959231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774456.727162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774456.983305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774457.239168:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239182:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774457.239188:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774457.239201:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239227:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774457.239280:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664571392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774457.239308:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239317:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774457.239319:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774457.239325:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239337:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774457.239349:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774457.239368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664571392:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774457.239368:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664571520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774457.239393:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239397:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774457.239399:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774457.239404:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239410:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239415:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774457.239421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774457.239437:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664571648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774457.239449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774457.239452:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774457.239456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774457.239459:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774457.239465:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239470:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239475:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774457.239480:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774457.239494:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe3b80 x1862032664571392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774457.239500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664571776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774457.239513:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774457.239517:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774457.239519:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664571392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774457.239523:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239528:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239534:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774457.239537:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775774457.239538:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774457.239546:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775774457.239550:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664571904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774457.239552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664571392:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774457.239565:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239569:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774457.239570:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774457.239574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239576:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239578:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774457.239581:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774457.239584:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774457.239592:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664572032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774457.239601:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774457.239603:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774457.239605:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774457.239609:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774457.239612:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16224 00000100:00080000:1.0:1775774457.239617:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774457.239619:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774457.239623:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774457.239625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe1f80 x1862032664571520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239639:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664571520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774457.239642:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664572160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774457.239667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239673:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774457.239685:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774457.239708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774457.239718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe0a80 x1862032664571648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664571648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774457.239733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774457.239741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774457.239757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774457.239763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe0e00 x1862032664571776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664571776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774457.239781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774457.239788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774457.239799:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774457.239804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe1c00 x1862032664571904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664571904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774457.239816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664571904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774457.239831:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774457.239842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774457.239848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe1500 x1862032664572032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664572032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774457.239860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774457.239869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774457.239879:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774457.239884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774457/real 1775774457] req@ffff982981fe1180 x1862032664572160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774457.239891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664572160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774457.239912:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774457.239915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774457.239919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774457.751162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774458.007294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774458.775146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774459.031271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774459.799166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774460.055278:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774460.823214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774461.079255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774461.847194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774462.103235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774462.359183:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359193:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774462.359197:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774462.359208:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359222:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359229:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774462.359275:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664572288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774462.359303:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359307:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774462.359309:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774462.359314:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359318:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359323:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774462.359334:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774462.359348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664572416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774462.359357:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664572288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774462.359358:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359362:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774462.359364:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774462.359367:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359371:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359375:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774462.359379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774462.359389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664572544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774462.359398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774462.359403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774462.359407:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359419:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359423:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774462.359428:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774462.359437:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774462.359437:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664572672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774462.359457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774462.359463:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774462.359466:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359469:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359473:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774462.359478:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774462.359481:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe1180 x1862032664572288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774462.359487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664572800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774462.359496:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774462.359501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774462.359502:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664572288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774462.359505:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359509:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359512:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774462.359514:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775774462.359517:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774462.359518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359524:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664572288:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774462.359526:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664572928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774462.359544:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774462.359547:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774462.359549:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774462.359551:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572416:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774462.359552:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16229 00000100:00080000:1.0:1775774462.359557:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774462.359561:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774462.359565:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774462.359574:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774462.359576:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664573056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774462.359587:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe1500 x1862032664572416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359600:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664572416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359609:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359612:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359615:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774462.359620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774462.359642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774462.359652:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe1c00 x1862032664572544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664572544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359672:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774462.359676:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774462.359690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774462.359697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe0e00 x1862032664572672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664572672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359711:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359713:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774462.359717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774462.359727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774462.359732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe0a80 x1862032664572800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664572800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664572800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774462.359751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774462.359761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774462.359766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe1f80 x1862032664572928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359773:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664572928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664572928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774462.359797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774462.359844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774462.359856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774462/real 1775774462] req@ffff982981fe3b80 x1862032664573056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774462.359869:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664573056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774462.359879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774462.359883:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774462.359886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774462.871202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774463.127268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774463.895149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774464.151263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774464.919183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774465.175280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774465.943190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774466.199253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774466.967161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774467.223270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774467.479242:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479250:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774467.479254:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479262:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479273:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774467.479278:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774467.479310:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664573184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774467.479332:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774467.479337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479340:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774467.479345:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774467.479353:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774467.479361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664573312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774467.479368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774467.479372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479375:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479377:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774467.479382:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664573184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774467.479383:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774467.479389:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774467.479396:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664573440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774467.479407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774467.479413:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479417:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479422:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774467.479426:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774467.479432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774467.479444:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774467.479448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664573568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774467.479460:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479464:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774467.479466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479470:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479475:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774467.479479:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774467.479483:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774467.479485:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b5c00 x1862032664573184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774467.479495:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664573696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774467.479502:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664573184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774467.479507:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479517:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774467.479520:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774467.479521:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479527:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774467.479527:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00100000:0.0:1775774467.479535:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664573184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774467.479535:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774467.479544:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479545:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573312:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775774467.479554:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774467.479568:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664573824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774467.479573:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774467.479582:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774467.479584:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774467.479586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b6a00 x1862032664573312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774467.479587:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774467.479596:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16234 00000100:00080000:2.0:1775774467.479604:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774467.479605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664573312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774467.479611:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774467.479615:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775774467.479619:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774467.479620:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479626:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774467.479639:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664573952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774467.479649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774467.479667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b6d80 x1862032664573440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774467.479686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664573440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774467.479692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774467.479702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774467.479713:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b5180 x1862032664573568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774467.479756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664573568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774467.479762:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774467.479770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774467.479784:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479790:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b4a80 x1862032664573696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774467.479798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664573696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774467.479803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664573696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774467.479810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774467.479820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479826:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b5f80 x1862032664573824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774467.479833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664573824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774467.479838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774467.479854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774467.479864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774467.479870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774467/real 1775774467] req@ffff9829764b4700 x1862032664573952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774467.479877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664573952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774467.479882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774467.479883:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774467.479885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664573952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774467.991146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774468.247231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774469.015181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774469.271232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774470.039257:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774470.295215:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774471.063196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774471.319306:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774472.087202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774472.343280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774472.599179:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599190:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774472.599196:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599207:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599220:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599227:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774472.599282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664574080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774472.599310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599313:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774472.599315:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599318:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599321:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774472.599334:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774472.599347:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664574208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774472.599356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664574080:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774472.599358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599366:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774472.599368:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599372:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599374:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599377:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774472.599380:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774472.599389:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664574336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774472.599409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774472.599415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599420:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599425:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599429:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774472.599436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774472.599436:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774472.599454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664574464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774472.599465:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774472.599472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599476:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2d180 x1862032664574080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774472.599490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774472.599504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664574592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774472.599535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774472.599538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774472.599539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599542:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599545:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599550:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599550:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664574080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774472.599556:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774472.599567:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664574720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774472.599567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599575:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774472.599580:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774472.599583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664574080:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774472.599584:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774472.599587:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774472.599593:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16240 00000100:00080000:0.0:1775774472.599595:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774472.599598:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774472.599601:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774472.599607:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664574848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774472.599614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774472.599656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2df80 x1862032664574208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664574208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774472.599693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774472.599706:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599712:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2d500 x1862032664574336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664574336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599735:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774472.599738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774472.599749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2fb80 x1862032664574464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599762:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664574464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774472.599775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774472.599785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2f100 x1862032664574592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664574592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664574592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774472.599810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664574720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774472.599820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2ca80 x1862032664574720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664574720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664574720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774472.599856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664574848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774472.599875:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774472.599886:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774472/real 1775774472] req@ffff982974b2c380 x1862032664574848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774472.599899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664574848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774472.599909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774472.599912:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774472.599916:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664574848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774473.111199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774473.367278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774474.135203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774474.391269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774475.159197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774475.415274:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774476.183170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774476.439277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774477.207246:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774477.463334:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774477.719208:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774477.719226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719239:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16245 00000100:00080000:2.0:1775774477.719256:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719265:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774477.719334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664574976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774477.719370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774477.719377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16245 00000100:00080000:2.0:1775774477.719388:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774477.719406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774477.719421:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664574976:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774477.719427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664575104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774477.719440:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719444:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774477.719447:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719451:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16245 00000100:00080000:2.0:1775774477.719455:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719460:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774477.719466:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774477.719481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664575232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774477.719494:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775774477.719500:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774477.719502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16245 00000100:00080000:2.0:1775774477.719514:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719520:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774477.719526:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774477.719557:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b6680 x1862032664574976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774477.719570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664575360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774477.719578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664574976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.719622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774477.719623:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719631:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774477.719633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774477.719635:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774477.719642:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664574976:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774477.719644:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16245 00000100:00080000:2.0:1775774477.719669:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719675:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774477.719683:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774477.719699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774477.719701:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664575488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774477.719718:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774477.719722:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774477.719725:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719731:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16245 00000800:00000100:3.0:1775774477.719733:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774477.719740:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719747:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774477.719753:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774477.719758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b5500 x1862032664575104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774477.719771:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664575616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774477.719772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664575104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.719788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.719795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.719799:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575104:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774477.719807:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774477.719808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575232:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774477.719814:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774477.719817:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774477.719823:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16245 00000800:00000100:3.0:1775774477.719829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774477.719830:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774477.719838:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774477.719840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b7100 x1862032664575232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774477.719849:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774477.719862:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664575232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774477.719870:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664575744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774477.719871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.719877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.719883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774477.719893:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774477.719909:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774477.719915:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b7b80 x1862032664575360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774477.719924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664575360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.719930:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.719933:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.719935:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774477.719949:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774477.719959:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774477.719965:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b4e00 x1862032664575488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774477.719972:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664575488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.719982:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.719985:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.719987:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664575488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774477.719990:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664575616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774477.719999:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774477.720004:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b4000 x1862032664575616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774477.720011:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664575616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.720017:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.720019:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.720021:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664575616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774477.720024:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664575744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774477.720033:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774477.720039:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774477/real 1775774477] req@ffff9829764b6680 x1862032664575744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774477.720046:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664575744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774477.720051:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774477.720052:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774477.720055:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664575744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774478.231206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774478.487248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774479.255196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774479.511291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774480.279195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774480.535287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774481.303170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774481.559283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774482.327202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774482.583261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774482.839221:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774482.839236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839251:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839265:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839272:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774482.839325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664575872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774482.839368:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839373:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839376:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839380:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774482.839391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774482.839404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664576000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839419:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775774482.839420:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664575872:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774482.839421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839428:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839431:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839433:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774482.839436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774482.839444:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664576128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839466:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774482.839471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839475:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839478:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839481:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774482.839486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774482.839495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664576256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839505:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774482.839506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839512:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775774482.839513:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774482.839517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774482.839522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774482.839531:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664576384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839538:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839541:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774482.839542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839545:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839549:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839552:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774482.839555:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774482.839556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b4700 x1862032664575872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774482.839565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664576512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774482.839574:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774482.839578:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775774482.839579:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664575872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774482.839581:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774482.839589:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16250 00000100:00080000:2.0:1775774482.839594:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774482.839598:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774482.839598:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775774482.839606:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774482.839607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664575872:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774482.839617:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664576640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774482.839639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774482.839659:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b5f80 x1862032664576000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664576000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839688:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774482.839692:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774482.839704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b4a80 x1862032664576128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839718:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664576128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774482.839732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774482.839743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b5180 x1862032664576256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664576256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839762:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774482.839769:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774482.839779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b6d80 x1862032664576384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664576384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774482.839805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664576512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774482.839815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b6a00 x1862032664576512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664576512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839843:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664576512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774482.839850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664576640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774482.839861:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774482.839867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774482/real 1775774482] req@ffff9829764b5c00 x1862032664576640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774482.839874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664576640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774482.839880:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774482.839882:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774482.839884:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664576640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774483.351192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774483.607278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774484.375208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774484.631306:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774485.399166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774485.655233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774486.423147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774486.679253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774487.447179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774487.703201:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774487.959164:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774487.959172:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774487.959178:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774487.959187:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00080000:2.0:1775774487.959199:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959206:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774487.959248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664576768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774487.959281:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774487.959286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774487.959287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774487.959292:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00080000:2.0:1775774487.959296:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959301:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774487.959312:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774487.959327:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664576896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774487.959333:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664576768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774487.959339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774487.959343:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774487.959344:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774487.959349:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00080000:2.0:1775774487.959354:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959390:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774487.959397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774487.959408:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774487.959414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664577024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774487.959434:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774487.959438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774487.959441:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774487.959445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b5c00 x1862032664576768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774487.959449:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00080000:2.0:1775774487.959458:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959464:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959469:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664576768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774487.959470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774487.959485:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775774487.959486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664577152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774487.959495:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774487.959500:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774487.959504:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664576768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774487.959505:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774487.959509:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774487.959517:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00080000:2.0:1775774487.959520:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959523:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774487.959528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774487.959539:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664577280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774487.959543:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576896:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774487.959561:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774487.959565:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774487.959568:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774487.959574:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16255 00000800:00000100:3.0:1775774487.959578:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774487.959583:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774487.959590:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959596:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b6a00 x1862032664576896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774487.959597:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774487.959613:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664577408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774487.959617:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664576896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774487.959628:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774487.959634:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774487.959635:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774487.959640:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774487.959643:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774487.959651:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16255 00000100:00100000:3.0:1775774487.959652:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664576896:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774487.959661:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774487.959662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577024:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774487.959672:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774487.959678:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774487.959684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774487.959698:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664577536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774487.959698:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b6d80 x1862032664577024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774487.959720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664577024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774487.959730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774487.959733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774487.959736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774487.959741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774487.959754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b5180 x1862032664577152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774487.959768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664577152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774487.959772:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774487.959774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774487.959777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774487.959789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774487.959799:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b4a80 x1862032664577280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774487.959812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664577280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774487.959817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774487.959818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774487.959820:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774487.959824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664577408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774487.959833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959839:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b5f80 x1862032664577408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774487.959846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664577408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774487.959851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774487.959852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774487.959854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664577408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774487.959857:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664577536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774487.959866:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774487.959872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774487/real 1775774487] req@ffff9829764b4700 x1862032664577536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774487.959879:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664577536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774487.959887:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774487.959890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774487.959894:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664577536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774488.471187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774488.727256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774489.495143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774489.751235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774490.519213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774490.775252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774491.543223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774491.799209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774492.567205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774492.823225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774493.079168:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079177:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774493.079181:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079203:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079212:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774493.079255:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664577664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774493.079303:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774493.079310:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079314:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079318:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079321:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079331:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774493.079356:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664577792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774493.079359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664577664:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774493.079372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774493.079381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079395:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079398:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774493.079422:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664577920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774493.079432:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774493.079437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079446:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774493.079469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664578048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774493.079481:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774493.079487:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079492:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079497:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775774493.079502:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774493.079509:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079518:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774493.079533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664578176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774493.079542:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079545:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774493.079547:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079550:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079554:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079560:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079566:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774493.079578:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664578304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774493.079580:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b6680 x1862032664577664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774493.079614:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774493.079630:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774493.079642:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774493.079646:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16260 00000100:00080000:2.0:1775774493.079650:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774493.079653:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774493.079657:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774493.079658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664577664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.079679:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775774493.079681:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664578432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774493.079691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.079709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664577664:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774493.079753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774493.079784:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.079800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b4000 x1862032664577792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.079815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664577792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.079825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.079829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.079833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774493.079841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774493.079862:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.079873:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b4e00 x1862032664577920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.079886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664577920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.079895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.079899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.079903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664577920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774493.079911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774493.079933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.079943:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b7b80 x1862032664578048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.079956:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664578048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.079966:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.079970:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.079973:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774493.079980:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774493.080002:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.080014:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b7100 x1862032664578176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.080027:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664578176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.080035:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.080039:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.080042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774493.080048:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664578304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774493.080101:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.080113:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b5500 x1862032664578304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.080125:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664578304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.080134:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.080137:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.080141:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664578304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774493.080147:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664578432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774493.080165:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774493.080175:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774493/real 1775774493] req@ffff9829764b4700 x1862032664578432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774493.080185:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664578432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774493.080192:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774493.080195:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774493.080199:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664578432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774493.591233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774493.847243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774494.615179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774494.871328:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774495.639243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774495.895261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774496.663219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774496.919339:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774497.687212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774497.943277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774498.199119:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199127:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774498.199131:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199140:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199149:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199155:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774498.199195:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664578560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774498.199215:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199218:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774498.199219:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199223:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199226:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199229:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774498.199238:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774498.199249:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664578688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774498.199255:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664578560:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774498.199272:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774498.199275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199283:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774498.199294:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774498.199305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664578816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774498.199312:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774498.199316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199318:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199321:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199323:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774498.199327:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774498.199334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664578944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774498.199341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774498.199345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199350:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199354:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199359:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774498.199365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774498.199368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774498.199380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664579072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774498.199394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774498.199401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00080000:2.0:1775774498.199411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774498.199418:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578816:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775774498.199422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774498.199437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664579200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774498.199444:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774498.199447:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774498.199449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774498.199454:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16265 00000100:00100000:3.0:1775774498.199455:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578944:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775774498.199458:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774498.199465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774498.199469:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774498.199477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664579328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774498.199501:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774498.199527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664579200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774498.199548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664579328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774498.711163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774498.967250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774499.735203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774499.991255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774500.759157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774501.015212:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775774501.271444:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775774501.271483:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775774501.271497:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271530:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271536:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271541:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271551:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271556:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775774501.271564:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775774501.271579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b6a00 x1862032664579072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664579072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774501.271648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b5c00 x1862032664579200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664579200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271676:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271679:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664579200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774501.271689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b4700 x1862032664579328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664579328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271709:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664579328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774501.271723:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b4a80 x1862032664578688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664578688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271746:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774501.271756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b5180 x1862032664578816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664578816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271777:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774501.271791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b6d80 x1862032664578944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664578944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664578944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774501.271826:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774498/real 1775774501] req@ffff9829764b5f80 x1862032664578560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774501.271853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664578560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774501.271862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774501.271866:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774501.271870:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664578560:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774501.783222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774502.039258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774502.807154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774503.063202:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774503.319262:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319270:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774503.319273:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774503.319282:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319292:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319297:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774503.319327:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664579456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774503.319349:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319352:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774503.319353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774503.319356:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319358:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319362:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774503.319370:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774503.319384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664579584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774503.319394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774503.319401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774503.319409:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664579456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774503.319413:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319424:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319429:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774503.319435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774503.319449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664579712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774503.319459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774503.319464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774503.319469:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319473:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775774503.319482:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775774503.319491:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774503.319507:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664579840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774503.319517:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775774503.319524:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2f480 x1862032664579456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774503.319525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774503.319534:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319563:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774503.319567:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664579456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774503.319570:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774503.319585:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774503.319590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664579968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774503.319594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.319603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664579456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774503.319608:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774503.319617:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774503.319620:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774503.319623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579584:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774503.319628:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319634:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319638:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774503.319644:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774503.319656:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664580096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774503.319666:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775774503.319669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774503.319680:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774503.319683:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774503.319684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2e300 x1862032664579584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774503.319692:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16270 00000100:00080000:0.0:1775774503.319701:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774503.319707:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774503.319710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664579584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774503.319716:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774503.319725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.319730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774503.319736:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664580224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774503.319737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774503.319773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774503.319795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774503.319808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2e680 x1862032664579712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774503.319822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664579712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774503.319832:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.319836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.319840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774503.319846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774503.319867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774503.319878:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2c380 x1862032664579840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774503.319890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664579840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774503.319898:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.319901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.319905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774503.319911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774503.319928:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774503.320044:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2ca80 x1862032664579968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774503.320105:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664579968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774503.320114:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.320118:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.320121:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664579968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774503.320128:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774503.320150:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774503.320162:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2f100 x1862032664580096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774503.320174:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664580096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774503.320184:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.320188:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.320191:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774503.320197:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774503.320218:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774503.320230:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774503/real 1775774503] req@ffff982974b2fb80 x1862032664580224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774503.320242:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664580224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774503.320251:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774503.320256:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774503.320259:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774503.831185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774504.087304:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774504.855168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774505.111285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774505.879233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774506.135239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774506.903172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774507.159253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774507.927309:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774508.183267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774508.439196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774508.439223:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774508.439235:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439251:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439260:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774508.439309:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664580352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774508.439341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774508.439346:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774508.439350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439354:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439358:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774508.439383:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664580480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774508.439388:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664580352:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774508.439391:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774508.439398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774508.439402:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439404:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439406:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774508.439421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664580608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774508.439437:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439441:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774508.439443:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774508.439448:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439454:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439459:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775774508.439465:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439466:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774508.439483:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664580736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774508.439495:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774508.439502:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774508.439507:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b5500 x1862032664580352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774508.439511:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439518:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439526:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439532:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774508.439540:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664580352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774508.439546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664580864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774508.439564:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774508.439565:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439570:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774508.439573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774508.439573:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774508.439582:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00100000:3.0:1775774508.439583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664580352:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774508.439587:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439592:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774508.439602:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664580992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774508.439612:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774508.439616:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774508.439618:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774508.439620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774508.439627:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16275 00000100:00080000:2.0:1775774508.439633:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774508.439637:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774508.439644:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774508.439656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774508.439660:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664581120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774508.439672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b7100 x1862032664580480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664580480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439693:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774508.439701:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774508.439714:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774508.439728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b7b80 x1862032664580608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664580608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439750:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774508.439765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774508.439788:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774508.439797:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b4e00 x1862032664580736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439806:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664580736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774508.439820:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774508.439833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774508.439840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b4000 x1862032664580864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664580864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664580864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774508.439859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774508.439870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774508.439876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b6680 x1862032664580992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439883:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664580992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439888:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664580992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774508.439909:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664581120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774508.439925:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774508.439935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774508/real 1775774508] req@ffff9829764b5500 x1862032664581120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774508.439947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664581120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774508.439957:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774508.439960:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774508.439963:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664581120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774508.951197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774509.207194:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774509.975206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774510.231320:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774510.999203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774511.255281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774512.023152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774512.279241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774513.047183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774513.303252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774513.559191:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774513.559206:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774513.559216:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559234:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774513.559273:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664581248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774513.559306:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774513.559312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774513.559316:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559323:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774513.559333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774513.559344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664581376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774513.559355:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774513.559361:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774513.559367:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664581248:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774513.559368:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559379:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774513.559384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774513.559394:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664581504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774513.559412:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559416:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774513.559417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774513.559421:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559427:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559430:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774513.559436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774513.559439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774513.559449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664581632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774513.559460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559464:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774513.559466:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774513.559470:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559474:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559479:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559480:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9f100 x1862032664581248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774513.559485:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774513.559498:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664581760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774513.559508:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664581248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774513.559510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559519:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774513.559523:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774513.559531:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774513.559531:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16281 00000100:00080000:3.0:1775774513.559537:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774513.559540:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774513.559544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559549:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664581248:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774513.559551:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774513.559561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774513.559566:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664581888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774513.559577:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774513.559580:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774513.559582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774513.559589:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16281 00000800:00000100:0.0:1775774513.559590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774513.559595:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774513.559601:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774513.559606:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774513.559609:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9df80 x1862032664581376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774513.559619:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664582016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774513.559628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664581376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.559640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.559645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.559649:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774513.559677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774513.559697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9fb80 x1862032664581504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774513.559728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664581504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.559737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.559740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.559744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774513.559749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774513.559767:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9e300 x1862032664581632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774513.559787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664581632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.559795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.559798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.559802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774513.559808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774513.559822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9d880 x1862032664581760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774513.559841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664581760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.559848:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.559851:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.559854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664581760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774513.559860:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664581888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774513.559879:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9c700 x1862032664581888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774513.559901:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664581888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.559910:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.559947:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.559950:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664581888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774513.559957:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774513.559980:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774513.559993:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774513/real 1775774513] req@ffff982980f9ce00 x1862032664582016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774513.560005:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664582016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774513.560020:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774513.560024:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774513.560027:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774514.071175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774514.327276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774515.095223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774515.351265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774516.119176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774516.375251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774517.143229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774517.399277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774518.167202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774518.423268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774518.679192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774518.679208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774518.679213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679223:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00080000:3.0:1775774518.679237:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774518.679244:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774518.679284:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664582144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774518.679321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774518.679325:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774518.679327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679331:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00080000:3.0:1775774518.679335:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774518.679340:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774518.679350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774518.679365:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664582272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774518.679377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774518.679380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664582144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774518.679383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774518.679386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679393:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00080000:3.0:1775774518.679397:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774518.679401:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774518.679407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774518.679420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664582400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774518.679431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774518.679435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774518.679437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00080000:3.0:1775774518.679448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774518.679452:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774518.679457:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774518.679469:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774518.679487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664582528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774518.679491:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9c000 x1862032664582144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774518.679525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664582144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774518.679532:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774518.679542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774518.679543:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00080000:0.0:1775774518.679548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774518.679549:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774518.679553:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664582144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774518.679558:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774518.679565:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774518.679576:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774518.679581:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664582656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774518.679597:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774518.679601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774518.679605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774518.679609:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679615:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00100000:0.0:1775774518.679616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9f800 x1862032664582272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679623:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774518.679634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664582272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679639:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774518.679650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775774518.679650:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774518.679655:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774518.679663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774518.679669:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664582784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774518.679680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774518.679683:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774518.679688:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774518.679691:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9d180 x1862032664582400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679692:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774518.679703:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16286 00000100:00100000:0.0:1775774518.679708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664582400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774518.679711:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774518.679718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774518.679720:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774518.679728:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774518.679731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679735:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774518.679740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774518.679746:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664582912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774518.679757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774518.679768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9ce00 x1862032664582528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774518.679776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664582528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774518.679783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774518.679785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679787:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774518.679799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774518.679809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774518.679815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9c700 x1862032664582656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774518.679849:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664582656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774518.679856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774518.679858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664582656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774518.679865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774518.679879:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774518.679887:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9d880 x1862032664582784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774518.679894:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664582784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774518.679899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774518.679901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774518.679907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774518.679917:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774518.679922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774518/real 1775774518] req@ffff982980f9e300 x1862032664582912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774518.679937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664582912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774518.679942:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774518.679944:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774518.679946:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664582912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774519.191181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774519.447200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774520.215164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774520.471233:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774521.239176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774521.495295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774522.263149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774522.519215:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774523.287233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774523.543299:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774523.799218:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774523.799232:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799241:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799254:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774523.799261:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774523.799300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664583040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774523.799328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799332:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774523.799334:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799339:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799343:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774523.799346:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774523.799355:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774523.799365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664583168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774523.799373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799376:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774523.799377:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799380:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799383:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774523.799387:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799389:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664583040:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775774523.799392:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774523.799413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664583296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774523.799420:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799423:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774523.799424:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799427:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799430:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774523.799432:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774523.799435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774523.799443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664583424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774523.799449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799455:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774523.799456:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799459:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799461:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774523.799463:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774523.799463:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774523.799469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774523.799476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664583552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774523.799489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799491:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774523.799493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774523.799496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:2.0:1775774523.799498:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774523.799501:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b5f80 x1862032664583040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774523.799504:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774523.799511:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774523.799523:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664583680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774523.799528:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664583040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774523.799544:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774523.799548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774523.799549:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774523.799550:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774523.799556:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16291 00000100:00080000:0.0:1775774523.799557:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774523.799559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774523.799564:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664583040:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774523.799570:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799573:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583168:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775774523.799576:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774523.799587:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664583808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774523.799601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799612:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b6d80 x1862032664583168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664583168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799627:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774523.799650:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774523.799662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b5180 x1862032664583296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799676:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664583296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774523.799688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774523.799698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b4a80 x1862032664583424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799711:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664583424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774523.799723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774523.799772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b4700 x1862032664583552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664583552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664583552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774523.799800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664583680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774523.799811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799817:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b5c00 x1862032664583680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664583680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664583680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774523.799836:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664583808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774523.799845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774523.799850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774523/real 1775774523] req@ffff9829764b6a00 x1862032664583808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774523.799857:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664583808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774523.799862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774523.799864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774523.799865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664583808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774524.311176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774524.567218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774525.335242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774525.591278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774526.359223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774526.615251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774527.383162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774527.639260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774528.407178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774528.663243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774528.919189:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919197:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774528.919201:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919210:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919220:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774528.919225:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774528.919261:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664583936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774528.919283:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919286:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774528.919287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919290:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919292:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774528.919295:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919302:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774528.919323:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664584064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774528.919331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919334:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774528.919335:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919339:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00100000:0.0:1775774528.919340:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664583936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774528.919342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774528.919349:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919352:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774528.919361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664584192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774528.919367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774528.919371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919374:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919377:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774528.919380:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919384:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774528.919390:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664584320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774528.919396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919400:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774528.919402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919406:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919410:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774528.919414:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774528.919417:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774528.919438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664584448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774528.919448:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9fb80 x1862032664583936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774528.919451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774528.919464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919468:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774528.919473:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664583936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774528.919480:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919488:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774528.919491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919495:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774528.919499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664584576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774528.919501:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664583936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774528.919516:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774528.919519:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774528.919521:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774528.919524:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16296 00000100:00080000:3.0:1775774528.919526:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774528.919528:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774528.919531:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774528.919538:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774528.919540:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664584704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774528.919575:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9df80 x1862032664584064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919595:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664584064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919603:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919606:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774528.919610:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774528.919623:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919629:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9f100 x1862032664584192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919636:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664584192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919641:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774528.919671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774528.919687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9c380 x1862032664584320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664584320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919712:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774528.919715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774528.919725:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9ea00 x1862032664584448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664584448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774528.919748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664584576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774528.919758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9e680 x1862032664584576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664584576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664584576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774528.919794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664584704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774528.919804:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774528.919810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774528/real 1775774528] req@ffff982980f9d500 x1862032664584704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774528.919818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664584704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774528.919823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774528.919825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774528.919827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664584704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774529.431183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774529.687261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774530.455185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774530.711246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774531.479189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774531.735225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774532.503174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774532.759257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774533.527165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774533.783262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774534.039211:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039220:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774534.039225:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039234:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039244:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774534.039250:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774534.039283:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664584832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774534.039314:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039317:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774534.039319:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039322:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039325:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774534.039328:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039337:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774534.039342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664584832:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774534.039349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664584960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774534.039356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774534.039360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039362:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039364:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774534.039366:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774534.039378:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664585088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774534.039388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039391:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774534.039392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039394:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039396:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774534.039399:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039402:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774534.039410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664585216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774534.039414:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774534.039421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039424:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774534.039426:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039430:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039433:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774534.039435:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774534.039446:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664585344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774534.039450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2d500 x1862032664584832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774534.039465:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039473:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774534.039475:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664584832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774534.039476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039491:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774534.039493:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774534.039494:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039501:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774534.039502:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039507:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664584832:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774534.039512:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664585472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774534.039518:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774534.039521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774534.039522:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774534.039524:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16301 00000100:00080000:0.0:1775774534.039527:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774534.039530:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584960:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774534.039532:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774534.039539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774534.039548:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664585600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774534.039571:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2df80 x1862032664584960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039601:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664584960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664584960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774534.039615:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774534.039627:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039633:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2d180 x1862032664585088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039640:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664585088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039648:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039649:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774534.039653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774534.039663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2fb80 x1862032664585216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039676:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664585216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774534.039687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774534.039697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039702:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2f100 x1862032664585344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664585344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774534.039721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664585472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774534.039730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2ca80 x1862032664585472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664585472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664585472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774534.039763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664585600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774534.039773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774534.039778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774534/real 1775774534] req@ffff982974b2c380 x1862032664585600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774534.039785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664585600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774534.039790:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774534.039791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774534.039793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664585600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774534.551196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774534.807196:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774535.575160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774535.831504:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774536.599211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774536.855310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774537.623160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774537.879289:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774538.647224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774538.903316:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774539.159284:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159294:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774539.159298:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159307:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159318:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159324:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774539.159358:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664585728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774539.159388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159392:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774539.159394:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159399:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159403:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159408:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774539.159419:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774539.159435:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664585856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774539.159449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159455:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775774539.159457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664585728:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774539.159458:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159483:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159486:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159489:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774539.159495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774539.159512:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664585984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774539.159519:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159522:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774539.159524:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159527:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159529:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159532:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774539.159535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774539.159543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664586112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774539.159548:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159551:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774539.159553:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159556:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159558:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159560:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774539.159563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774539.159569:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774539.159575:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664586240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774539.159582:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159585:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774539.159587:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159592:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159596:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159599:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774539.159603:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774539.159611:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664586368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774539.159618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe0380 x1862032664585728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774539.159624:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774539.159640:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774539.159647:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774539.159655:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16306 00000100:00080000:1.0:1775774539.159661:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774539.159665:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774539.159669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664585728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775774539.159672:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774539.159690:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664586496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774539.159692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.159709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.159716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664585728:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774539.159753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774539.159781:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.159795:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe0000 x1862032664585856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.159807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664585856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.159816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.159820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.159823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774539.159829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774539.159847:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.159857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe1880 x1862032664585984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.159868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664585984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.159877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.159880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.159883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664585984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774539.159890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774539.159908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.159918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe2300 x1862032664586112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.159929:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664586112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.159936:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.159939:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.159942:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774539.159948:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774539.159965:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.159976:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe2680 x1862032664586240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.159987:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664586240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.159995:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.159998:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.160001:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774539.160007:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664586368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774539.160027:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.160038:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe2a00 x1862032664586368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.160049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664586368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.160057:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.160061:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.160091:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664586368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774539.160097:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664586496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774539.160117:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774539.160129:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774539/real 1775774539] req@ffff982981fe3480 x1862032664586496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774539.160141:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664586496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774539.160150:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774539.160153:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774539.160157:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664586496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774539.671128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774539.927232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774540.695176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774540.951289:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774541.719206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774541.975288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774542.743175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774542.999275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774543.767181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774544.023249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774544.279188:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279198:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774544.279204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279215:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279231:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279246:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774544.279295:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664586624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774544.279331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774544.279338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279343:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279347:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774544.279393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664586752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774544.279403:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279406:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774544.279409:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279415:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00100000:0.0:1775774544.279417:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664586624:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774544.279419:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279429:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774544.279455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664586880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774544.279468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774544.279473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279476:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279481:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279484:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279489:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774544.279502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664587008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774544.279515:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774544.279524:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774544.279529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279533:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279538:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279542:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279548:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774544.279559:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664587136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774544.279566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9ed80 x1862032664586624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774544.279571:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279591:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774544.279598:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279611:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279616:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774544.279623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664586624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774544.279626:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279641:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774544.279644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774544.279655:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664587264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774544.279661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664586624:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774544.279682:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774544.279685:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774544.279688:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774544.279692:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16311 00000100:00080000:3.0:1775774544.279694:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774544.279698:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774544.279702:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774544.279704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774544.279722:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664587392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774544.279736:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9f480 x1862032664586752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664586752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774544.279776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774544.279788:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9dc00 x1862032664586880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279806:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664586880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664586880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774544.279819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774544.279830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9d500 x1862032664587008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664587008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279848:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774544.279855:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774544.279865:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9e680 x1862032664587136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664587136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279889:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279891:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774544.279894:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664587264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774544.279903:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9ea00 x1862032664587264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664587264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279920:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279922:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664587264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774544.279936:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664587392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774544.279947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774544.279952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774544/real 1775774544] req@ffff982980f9c380 x1862032664587392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774544.279959:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664587392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774544.279964:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774544.279966:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774544.279968:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664587392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774544.791210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774545.047285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774545.815144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774546.071213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774546.839136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774547.095268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774547.863193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774548.119231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774548.887149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774549.143277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774549.399152:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399172:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774549.399176:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399186:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399197:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399202:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774549.399264:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664587520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774549.399374:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399377:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774549.399378:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399382:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399385:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399389:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774549.399415:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664587648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774549.399423:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664587520:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774549.399438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774549.399441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399445:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399446:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399448:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399452:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774549.399460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664587776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774549.399467:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399469:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774549.399476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399479:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399481:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399484:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399488:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774549.399497:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664587904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774549.399504:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774549.399508:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399511:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774549.399512:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399515:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399519:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399522:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399526:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774549.399534:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664588032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774549.399541:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9f100 x1862032664587520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774549.399541:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399549:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774549.399552:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399561:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399564:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399567:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774549.399572:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664587520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774549.399579:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664588160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774549.399583:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399591:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774549.399591:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774549.399598:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774549.399601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664587520:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774549.399601:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774549.399608:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16316 00000100:00080000:3.0:1775774549.399613:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774549.399616:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774549.399619:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774549.399625:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664588288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774549.399637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774549.399663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9df80 x1862032664587648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664587648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774549.399698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774549.399710:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399716:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9fb80 x1862032664587776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664587776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774549.399737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774549.399747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9e300 x1862032664587904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664587904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664587904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774549.399773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774549.399783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399809:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9d880 x1862032664588032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664588032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399827:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774549.399834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664588160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774549.399847:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9c700 x1862032664588160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399882:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664588160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399892:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399894:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664588160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774549.399914:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664588288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774549.399948:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774549.399956:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774549/real 1775774549] req@ffff982980f9ce00 x1862032664588288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774549.399965:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664588288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774549.399972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774549.399975:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774549.399977:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664588288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774549.911198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774550.167361:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774550.935195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774551.191322:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774551.959279:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774552.215375:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774552.983181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774553.239402:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774554.007248:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774554.263406:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774554.519186:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774554.519234:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519245:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519258:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519266:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774554.519305:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664588416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774554.519332:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519335:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774554.519337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519340:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519345:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519349:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519357:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774554.519366:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664588416:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774554.519369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664588544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774554.519381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774554.519385:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519388:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519394:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519397:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774554.519406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664588672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774554.519413:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774554.519416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519419:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519421:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519423:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519427:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774554.519438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664588800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774554.519440:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774554.519448:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519451:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774554.519452:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519455:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519459:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519462:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519466:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774554.519472:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664588928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774554.519479:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519483:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774554.519483:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2e680 x1862032664588416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774554.519487:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774554.519495:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519501:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774554.519506:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519509:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774554.519515:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664588416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774554.519518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664589056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774554.519550:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774554.519552:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774554.519554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774554.519554:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774554.519562:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16322 00000100:00080000:0.0:1775774554.519566:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774554.519567:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774554.519572:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774554.519575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774554.519577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664588416:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774554.519584:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664589184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774554.519609:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774554.519643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519652:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2e300 x1862032664588544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519662:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664588544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774554.519678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774554.519690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2f480 x1862032664588672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519714:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664588672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519732:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774554.519736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774554.519754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2c380 x1862032664588800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664588800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774554.519795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774554.519805:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2ca80 x1862032664588928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664588928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519827:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664588928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774554.519832:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774554.519842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2f100 x1862032664589056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664589056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519864:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519866:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774554.519871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774554.519881:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774554.519886:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774554/real 1775774554] req@ffff982974b2fb80 x1862032664589184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774554.519894:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664589184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774554.519899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774554.519900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774554.519903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774555.031165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774555.287324:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774556.055213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774556.311297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774557.079212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774557.335304:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774558.103203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774558.359285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774559.127224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774559.383294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774559.639184:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639198:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774559.639203:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639216:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639230:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639238:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774559.639286:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664589312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774559.639333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774559.639344:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639351:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639356:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639362:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774559.639373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774559.639386:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664589440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774559.639398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639402:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774559.639406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774559.639410:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664589312:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774559.639416:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639423:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639430:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774559.639437:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774559.639454:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664589568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774559.639466:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639470:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774559.639472:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639477:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639481:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639485:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774559.639491:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774559.639506:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774559.639506:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664589696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774559.639523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774559.639529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639537:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639543:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639553:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774559.639554:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9d180 x1862032664589312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774559.639570:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774559.639586:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664589312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774559.639587:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664589824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774559.639649:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774559.639653:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774559.639655:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774559.639666:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775774559.639667:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664589312:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774559.639674:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639683:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639690:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639695:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774559.639701:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774559.639702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774559.639719:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664589952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774559.639730:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774559.639733:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774559.639735:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774559.639739:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16327 00000100:00080000:3.0:1775774559.639743:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774559.639746:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774559.639759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774559.639760:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774559.639775:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664590080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774559.639786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9f800 x1862032664589440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.639802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664589440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.639809:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.639812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.639816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774559.639822:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774559.639839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774559.639847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9c000 x1862032664589568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.639855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664589568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.639861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.639863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.639866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774559.639870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774559.639882:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774559.639889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9ce00 x1862032664589696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.639897:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664589696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.639903:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.639905:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.639907:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774559.639912:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774559.639923:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774559.639930:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9c700 x1862032664589824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.639938:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664589824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.639943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.639945:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.639948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664589824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774559.639962:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774559.639974:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774559.639980:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9d880 x1862032664589952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.639989:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664589952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.639994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.639996:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.639998:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664589952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774559.640002:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774559.640013:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774559.640019:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774559/real 1775774559] req@ffff982980f9e300 x1862032664590080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774559.640027:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664590080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774559.640033:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774559.640036:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774559.640038:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774560.151201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774560.407301:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774561.175188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774561.431242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774562.199256:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774562.455317:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774563.223209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774563.479248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774564.247174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774564.503293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774564.759196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759210:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774564.759216:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759253:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759263:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774564.759311:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664590208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774564.759339:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774564.759346:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759353:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759356:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774564.759368:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774564.759385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664590336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774564.759401:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664590208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774564.759401:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759416:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774564.759418:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759423:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759428:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759432:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774564.759440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774564.759458:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664590464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774564.759471:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774564.759478:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759482:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759487:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759492:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774564.759498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774564.759518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664590592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774564.759533:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774564.759537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590336:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774564.759545:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774564.759549:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759557:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759561:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759565:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774564.759572:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774564.759581:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774564.759588:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664590720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774564.759597:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759600:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774564.759602:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774564.759608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590592:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775774564.759613:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759622:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759629:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774564.759634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590720:192.168.201.118@tcp:8:kworker.0 00000008:00100000:2.0:1775774564.759636:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774564.759669:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664590848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774564.759683:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774564.759686:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774564.759688:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774564.759693:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16332 00000100:00080000:2.0:1775774564.759701:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774564.759705:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774564.759711:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774564.759712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774564.759724:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664590976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774564.759762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774565.271196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774565.527231:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774566.295169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774566.551249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774567.319166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774567.575269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774567.831290:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774567.831335:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774567.831351:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831384:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831390:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831395:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831402:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831409:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774567.831415:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774567.831431:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b5f80 x1862032664590976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831453:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664590976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831466:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831471:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831476:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774567.831483:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b6d80 x1862032664590848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831494:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664590848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831507:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831510:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664590848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774567.831516:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b4a80 x1862032664590592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831527:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664590592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831535:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831539:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831549:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774567.831556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b5180 x1862032664590720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664590720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831579:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831582:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774567.831588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b5c00 x1862032664590336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664590336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831604:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831608:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774567.831617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b4700 x1862032664590464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664590464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831641:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664590464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774567.831648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774564/real 1775774567] req@ffff9829764b6a00 x1862032664590208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774567.831659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664590208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774567.831666:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774567.831670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774567.831673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664590208:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774568.343206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774568.599252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774569.367184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774569.623265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774569.879175:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879184:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774569.879188:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774569.879196:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879206:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879212:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774569.879243:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664591104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774569.879268:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879271:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774569.879272:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774569.879276:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879278:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879280:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879288:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774569.879295:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664591232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774569.879305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879309:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774569.879312:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774569.879316:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879320:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879325:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879330:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774569.879340:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664591104:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774569.879344:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664591360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774569.879357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879361:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774569.879363:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774569.879367:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879371:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879375:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774569.879395:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664591488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774569.879404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879408:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774569.879410:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774569.879416:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774569.879420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879426:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879431:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879437:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774569.879450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664591616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774569.879457:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2d180 x1862032664591104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774569.879466:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774569.879479:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774569.879484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879488:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774569.879492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664591104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774569.879496:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774569.879521:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774569.879522:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664591744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774569.879531:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879539:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664591104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774569.879542:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774569.879548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774569.879550:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774569.879552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591232:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774569.879557:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16337 00000100:00080000:0.0:1775774569.879563:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774569.879566:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774569.879571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774569.879584:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664591872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774569.879588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2df80 x1862032664591232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664591232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.879634:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.879637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774569.879665:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774569.879686:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2d500 x1862032664591360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664591360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.879716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.879720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774569.879730:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774569.879752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2fb80 x1862032664591488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664591488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.879784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.879786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774569.879795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774569.879816:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2f100 x1862032664591616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664591616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.879844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.879847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664591616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774569.879855:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664591744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774569.879873:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2ca80 x1862032664591744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664591744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.879900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.879904:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.879907:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664591744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774569.879913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664591872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774569.879966:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774569.879979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774569/real 1775774569] req@ffff982974b2c380 x1862032664591872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774569.879992:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664591872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774569.880141:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774569.880144:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774569.880147:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664591872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774570.391199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774570.647249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774571.415170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774571.671229:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774572.439148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774572.695287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774573.463186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774573.719245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774574.487154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774574.743280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774574.999192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774574.999207:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774574.999218:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:2.0:1775774574.999231:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774574.999238:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774574.999278:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664592000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774574.999302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774574.999305:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774574.999308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:2.0:1775774574.999311:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774574.999315:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999323:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774574.999333:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664592128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774574.999341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774574.999346:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774574.999347:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664592000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774574.999351:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:2.0:1775774574.999356:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774574.999359:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999362:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774574.999370:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664592256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774574.999377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774574.999381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774574.999383:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:2.0:1775774574.999385:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774574.999387:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774574.999397:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664592384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774574.999404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774574.999408:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774574.999411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16342 00000800:00000100:3.0:1775774574.999416:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774574.999418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774574.999423:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999427:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774574.999436:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664592512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774574.999443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774574.999446:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774574.999450:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:2.0:1775774574.999452:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774574.999453:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b5500 x1862032664592000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774574.999457:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774574.999475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664592640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774574.999479:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664592000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774574.999492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774574.999497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774574.999498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774574.999498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774574.999507:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16342 00000100:00080000:3.0:1775774574.999508:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774574.999510:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774574.999514:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664592000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774574.999517:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774574.999523:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774574.999535:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664592768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774574.999538:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774574.999562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999573:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b6680 x1862032664592128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999582:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664592128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999587:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999590:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999592:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774574.999595:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774574.999606:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b4000 x1862032664592256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664592256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999628:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774574.999636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774574.999646:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999651:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b4e00 x1862032664592384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664592384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999667:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774574.999670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774574.999679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b7b80 x1862032664592512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664592512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664592512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774574.999704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664592640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774574.999713:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b7100 x1862032664592640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999725:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664592640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664592640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774574.999747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664592768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774574.999757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774574.999763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774574/real 1775774574] req@ffff9829764b6a00 x1862032664592768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774574.999771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664592768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774574.999775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774574.999777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774574.999779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664592768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774575.511197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774575.767245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774576.535169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774576.791218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774577.559173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774577.815234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774578.583153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774578.839242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774579.607167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774579.863278:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774580.119213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119221:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774580.119225:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119233:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:3.0:1775774580.119243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119249:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774580.119282:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664592896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774580.119307:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119310:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774580.119311:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119315:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:3.0:1775774580.119318:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119321:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774580.119329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774580.119338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664593024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774580.119345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119349:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774580.119351:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774580.119352:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664592896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774580.119355:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:3.0:1775774580.119360:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119363:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774580.119366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774580.119373:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664593152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774580.119395:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119398:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774580.119399:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119402:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:3.0:1775774580.119405:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119408:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774580.119412:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774580.119420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664593280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774580.119426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774580.119430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119436:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16347 00000800:00000100:0.0:1775774580.119439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774580.119441:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774580.119448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774580.119455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664593408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774580.119461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119464:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774580.119466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:3.0:1775774580.119472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774580.119475:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9fb80 x1862032664592896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774580.119477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774580.119484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774580.119495:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664593536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774580.119501:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664592896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774580.119504:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774580.119512:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774580.119515:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774580.119522:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16347 00000100:00080000:0.0:1775774580.119523:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774580.119524:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774580.119530:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774580.119534:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775774580.119535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774580.119540:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664592896:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774580.119545:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664593664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774580.119582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774580.119600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9df80 x1862032664593024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664593024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774580.119649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774580.119664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9f100 x1862032664593152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664593152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119686:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774580.119710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774580.119724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9c380 x1862032664593280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664593280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119746:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774580.119796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774580.119811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9ea00 x1862032664593408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664593408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774580.119837:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664593536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774580.119846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9e680 x1862032664593536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664593536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664593536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774580.119869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664593664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774580.119878:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774580.119883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774580/real 1775774580] req@ffff982980f9d500 x1862032664593664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774580.119895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664593664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774580.119900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774580.119901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774580.119903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664593664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774580.631204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774580.887293:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774581.655228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774581.911313:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774582.679192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774582.935253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774583.703177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774583.959211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774584.727179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774584.983204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774585.239182:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239191:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774585.239195:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239204:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239214:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239219:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774585.239253:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664593792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774585.239276:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239279:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774585.239280:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239284:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239286:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239289:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774585.239297:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774585.239306:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664593920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774585.239314:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239316:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774585.239318:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00100000:3.0:1775774585.239323:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664593792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774585.239327:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774585.239336:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774585.239343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664594048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774585.239351:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774585.239357:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239361:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239364:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239369:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774585.239375:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774585.239389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664594176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774585.239399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239404:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774585.239408:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774585.239409:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774585.239416:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239422:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239426:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774585.239430:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774585.239439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664594304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774585.239447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774585.239451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239455:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239458:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239462:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2f480 x1862032664593792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774585.239465:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774585.239482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664594432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774585.239489:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664593792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774585.239491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774585.239498:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774585.239501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774585.239509:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16352 00000100:00080000:0.0:1775774585.239514:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774585.239517:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774585.239518:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775774585.239520:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774585.239524:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239529:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664593792:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774585.239531:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664594560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774585.239556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774585.239580:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2e300 x1862032664593920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664593920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239617:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239624:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664593920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774585.239628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774585.239649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239702:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2e680 x1862032664594048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664594048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774585.239738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774585.239755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2c380 x1862032664594176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664594176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774585.239783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774585.239793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2ca80 x1862032664594304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239806:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664594304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239817:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774585.239824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664594432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774585.239842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239864:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2f100 x1862032664594432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239883:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664594432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239890:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239892:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239894:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664594432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774585.239898:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664594560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774585.239914:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774585.239921:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774585/real 1775774585] req@ffff982974b2fb80 x1862032664594560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774585.239928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664594560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774585.239933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774585.239935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774585.239937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664594560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774585.751171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774586.007241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774586.775181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774587.032150:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774587.799163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774588.055279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774588.823131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774589.079254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774589.847183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774590.103184:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774590.359214:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359222:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774590.359227:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359235:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359246:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359252:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774590.359287:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664594688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359313:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774590.359315:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359318:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359320:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359323:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359330:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774590.359338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664594816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359344:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359347:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774590.359348:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359352:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359354:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774590.359362:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664594944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774590.359370:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359373:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359375:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359377:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359380:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774590.359385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664595072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359391:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359393:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774590.359394:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359396:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359398:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359401:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774590.359410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664595200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774590.359423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359427:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359434:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774590.359434:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664594688:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774590.359441:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774590.359462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664595328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774590.359473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774590.359477:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774590.359480:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774590.359484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16357 00000100:00080000:1.0:1775774590.359487:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774590.359489:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774590.359494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774590.359502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664595456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774590.359509:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359549:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe3480 x1862032664594688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664594688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359612:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359618:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359623:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664594688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774590.359631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774590.359658:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe2a00 x1862032664594816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664594816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359694:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774590.359708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774590.359728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe2680 x1862032664594944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664594944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664594944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774590.359772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774590.359792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe2300 x1862032664595072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664595072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774590.359835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774590.359856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe1880 x1862032664595200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664595200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359896:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774590.359928:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664595328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774590.359956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.359968:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe0000 x1862032664595328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.359980:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664595328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.359989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.359992:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.359995:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664595328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774590.360001:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664595456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774590.360021:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774590.360031:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774590/real 1775774590] req@ffff982981fe0380 x1862032664595456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774590.360050:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664595456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774590.360058:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774590.360061:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774590.360091:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664595456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774590.871176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774591.127251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774591.895175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774592.151235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774592.919191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775774593.175230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774593.943162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774594.199275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774594.967208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774595.223268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774595.479231:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479242:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774595.479247:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774595.479257:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479270:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479277:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774595.479314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664595584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774595.479346:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774595.479352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774595.479356:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479360:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479364:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774595.479375:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774595.479387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664595712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774595.479399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774595.479406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774595.479409:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664595584:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774595.479412:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479419:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479423:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774595.479428:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774595.479440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664595840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774595.479450:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774595.479456:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774595.479460:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479464:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479468:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774595.479473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774595.479497:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774595.479501:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664595968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774595.479513:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479518:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774595.479520:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774595.479524:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479529:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479533:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479537:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9dc00 x1862032664595584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774595.479540:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774595.479556:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664596096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774595.479565:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664595584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774595.479570:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479579:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774595.479583:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774595.479587:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774595.479589:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479595:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774595.479596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774595.479603:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479605:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664595584:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774595.479609:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774595.479621:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664596224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774595.479632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595712:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774595.479634:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774595.479641:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774595.479643:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774595.479648:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16362 00000100:00080000:3.0:1775774595.479653:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774595.479657:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774595.479664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774595.479665:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774595.479678:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664596352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774595.479680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9f480 x1862032664595712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.479710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664595712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.479719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.479725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.479730:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774595.479737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774595.479755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9ed80 x1862032664595840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.479776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664595840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.479785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.479788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.479792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774595.479797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774595.479813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479822:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9d500 x1862032664595968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.479833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664595968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.479842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.479845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.479849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664595968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774595.479854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774595.479870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9e680 x1862032664596096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.479891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664596096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.479899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.479902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.479905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774595.479923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664596224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774595.479943:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774595.479953:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9ea00 x1862032664596224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.479965:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664596224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.479974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.479977:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.479981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664596224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774595.479987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664596352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774595.480002:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774595.480010:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774595/real 1775774595] req@ffff982980f9c380 x1862032664596352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774595.480029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664596352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774595.480038:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774595.480041:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774595.480044:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664596352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774595.991145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774596.247259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774597.015217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774597.271337:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774598.039273:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774598.295284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774599.063243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774599.319293:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774600.087144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774600.343261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774600.599167:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599175:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774600.599180:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599188:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599205:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774600.599245:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664596480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599268:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599271:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774600.599272:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599275:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599277:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599280:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774600.599287:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774600.599297:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664596608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599304:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599307:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774600.599309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599315:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599321:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664596480:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775774600.599323:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774600.599335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664596736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774600.599345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599347:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599349:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599351:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774600.599355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774600.599364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664596864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599372:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774600.599373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599376:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599378:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599380:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774600.599383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774600.599388:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774600.599407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664596992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774600.599420:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599423:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599430:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774600.599434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774600.599440:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b4700 x1862032664596480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774600.599444:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664597120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774600.599470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774600.599472:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774600.599474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774600.599477:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16368 00000100:00080000:2.0:1775774600.599479:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774600.599483:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599484:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664596480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774600.599486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774600.599497:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664597248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774600.599501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599510:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599514:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664596480:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774600.599520:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774600.599541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599550:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b5c00 x1862032664596608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664596608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599565:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599567:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774600.599586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774600.599597:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599603:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b5180 x1862032664596736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599610:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664596736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599618:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774600.599623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774600.599633:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b4a80 x1862032664596864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599645:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664596864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774600.599662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774600.599672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b6d80 x1862032664596992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664596992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599692:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664596992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774600.599695:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664597120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774600.599705:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b5f80 x1862032664597120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664597120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664597120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774600.599729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664597248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774600.599739:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774600.599744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774600/real 1775774600] req@ffff9829764b6a00 x1862032664597248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774600.599750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664597248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774600.599756:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774600.599757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774600.599759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664597248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774601.111178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774601.367243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774602.135185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774602.391212:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774603.159162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774603.415234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774604.183276:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774604.439248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774605.207180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774605.463285:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774605.719221:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719229:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774605.719234:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719242:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719253:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719258:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774605.719293:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664597376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774605.719334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774605.719339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774605.719368:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664597504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774605.719375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774605.719380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774605.719398:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664597376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774605.719401:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664597632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774605.719415:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774605.719420:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719424:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719428:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719431:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774605.719450:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664597760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774605.719461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774605.719467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719471:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774605.719480:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774605.719484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719491:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774605.719505:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664597888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774605.719517:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719520:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775774605.719525:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b7100 x1862032664597376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774605.719525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719533:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774605.719558:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664597376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.719570:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.719575:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775774605.719583:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664598016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774605.719584:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664597376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774605.719599:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597504:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774605.719601:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774605.719609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774605.719612:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774605.719617:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16373 00000100:00080000:2.0:1775774605.719622:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774605.719626:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774605.719632:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774605.719638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774605.719650:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664598144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774605.719658:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b7b80 x1862032664597504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.719678:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664597504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.719689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.719693:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.719698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774605.719773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774605.719796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774605.719807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b4e00 x1862032664597632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.719820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664597632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.719829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.719833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.719836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774605.719842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774605.719862:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774605.719874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b4000 x1862032664597760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.719887:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664597760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.719895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.719899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.719902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774605.719907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774605.719925:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774605.719934:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b6680 x1862032664597888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.719946:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664597888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.719954:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.719958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.719961:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664597888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774605.719966:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774605.719982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774605.719990:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b5500 x1862032664598016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.720000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664598016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.720007:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.720010:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.720013:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774605.720018:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774605.720033:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774605.720040:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774605/real 1775774605] req@ffff9829764b6a00 x1862032664598144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774605.720050:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664598144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774605.720057:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774605.720060:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774605.720088:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774606.231193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774606.487239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774607.255212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774607.511252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774608.279175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774608.535286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774609.303179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774609.559285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774610.327176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774610.583259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774610.839134:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839142:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774610.839146:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839154:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839164:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839170:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774610.839203:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664598272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774610.839241:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839245:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774610.839246:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839250:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839253:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839256:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774610.839264:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774610.839276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664598400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774610.839284:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839287:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774610.839290:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774610.839293:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664598272:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774610.839295:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839302:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839306:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774610.839312:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774610.839325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664598528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774610.839336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774610.839341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839345:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839348:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839351:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774610.839355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774610.839361:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774610.839368:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664598656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774610.839374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774610.839379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839383:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839386:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774610.839390:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774610.839397:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664598784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774610.839403:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9f100 x1862032664598272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774610.839404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774610.839413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839416:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839418:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774610.839420:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774610.839424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774610.839430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664598912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774610.839437:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774610.839439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774610.839441:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664598272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774610.839442:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774610.839450:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16378 00000100:00080000:3.0:1775774610.839458:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774610.839460:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774610.839464:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774610.839468:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839473:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664598272:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774610.839473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774610.839491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664599040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774610.839499:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774610.839526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839540:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9df80 x1862032664598400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839554:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664598400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839566:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839570:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774610.839576:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774610.839593:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839601:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9fb80 x1862032664598528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664598528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774610.839623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774610.839634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9e300 x1862032664598656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839647:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664598656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839652:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774610.839659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774610.839669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839675:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9d880 x1862032664598784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664598784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664598784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774610.839694:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774610.839703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9c700 x1862032664598912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839715:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664598912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664598912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774610.839738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774610.839748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774610.839753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774610/real 1775774610] req@ffff982980f9ce00 x1862032664599040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774610.839760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664599040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774610.839765:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774610.839767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774610.839768:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774611.351155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774611.607227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774612.375203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774612.631295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774613.399138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774613.655282:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774614.423184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774614.679252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774615.447190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774615.703239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774615.959212:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774615.959226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959235:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959246:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774615.959285:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664599168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774615.959315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774615.959319:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959322:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959325:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959327:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774615.959336:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774615.959344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664599296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774615.959351:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959354:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774615.959356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959359:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959361:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959363:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774615.959366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774615.959376:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664599424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774615.959384:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664599168:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774615.959387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774615.959397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959400:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959401:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959403:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774615.959407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774615.959417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664599552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774615.959430:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774615.959435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959437:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959440:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959442:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774615.959445:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774615.959454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664599680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774615.959458:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774615.959465:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774615.959470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959474:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959477:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959480:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774615.959483:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774615.959491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664599808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774615.959495:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2d500 x1862032664599168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774615.959501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774615.959508:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774615.959511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774615.959517:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16383 00000100:00080000:0.0:1775774615.959520:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774615.959524:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959524:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664599168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774615.959530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774615.959539:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774615.959540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664599936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774615.959548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664599168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774615.959562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774615.959588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2df80 x1862032664599296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664599296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959624:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959631:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774615.959653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774615.959668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2d180 x1862032664599424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664599424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959694:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774615.959702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774615.959713:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2fb80 x1862032664599552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959725:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664599552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774615.959737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774615.959746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2f100 x1862032664599680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664599680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664599680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774615.959786:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774615.959805:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2ca80 x1862032664599808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959827:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664599808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959835:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774615.959841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774615.959855:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774615.959861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774615/real 1775774615] req@ffff982974b2c380 x1862032664599936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774615.959868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664599936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774615.959873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774615.959875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774615.959877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664599936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774616.471268:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774616.727324:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774617.495136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774617.751233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774618.519256:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774618.775288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774619.543139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774619.799223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774620.567215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774620.823301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774621.079203:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079217:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774621.079224:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079243:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079259:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079267:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774621.079313:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664600064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774621.079341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774621.079346:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079350:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079352:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079355:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774621.079374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664600192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774621.079381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774621.079385:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079387:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079391:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079394:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774621.079402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664600320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774621.079412:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079415:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774621.079417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079423:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079429:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774621.079462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664600448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774621.079464:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664600064:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774621.079491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079494:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774621.079496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079501:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079506:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079509:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079514:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774621.079527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664600576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774621.079534:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079537:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774621.079538:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079542:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079544:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774621.079558:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774621.079568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664600704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774621.079582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774621.079584:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774621.079586:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774621.079592:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16388 00000100:00080000:0.0:1775774621.079596:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774621.079599:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774621.079604:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774621.079604:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2e680 x1862032664600064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774621.079620:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664600832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774621.079627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664600064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664600064:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774621.079661:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774621.079678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2e300 x1862032664600192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664600192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774621.079709:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774621.079719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2f480 x1862032664600320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664600320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774621.079762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774621.079774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2c380 x1862032664600448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664600448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079796:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774621.079800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774621.079811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2ca80 x1862032664600576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664600576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664600576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774621.079836:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664600704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774621.079845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2f100 x1862032664600704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664600704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664600704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774621.079869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664600832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774621.079878:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774621.079884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774621/real 1775774621] req@ffff982974b2fb80 x1862032664600832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774621.079890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664600832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774621.079894:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774621.079896:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774621.079899:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664600832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774621.591241:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774621.847224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774622.615201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774622.871243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774623.639206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774623.895209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774624.663203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774624.919267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774625.687185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774625.943223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774626.199146:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199155:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774626.199159:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199168:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199180:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199188:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774626.199225:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664600960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774626.199255:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199259:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774626.199260:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199264:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199267:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199271:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774626.199280:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774626.199293:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664600960:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774626.199293:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664601088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774626.199310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199312:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774626.199314:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199323:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774626.199326:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774626.199335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664601216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774626.199343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774626.199347:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199349:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199352:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199354:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774626.199358:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774626.199369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664601344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774626.199370:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774626.199379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774626.199383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199387:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199389:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199391:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774626.199395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774626.199402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664601472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774626.199407:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9c000 x1862032664600960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774626.199415:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774626.199423:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774626.199426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199431:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:3.0:1775774626.199434:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199436:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774626.199439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774626.199441:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664600960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774626.199447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664601600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774626.199470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774626.199473:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774626.199473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774626.199475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774626.199482:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16393 00000100:00080000:0.0:1775774626.199485:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774626.199486:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774626.199492:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664600960:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774626.199497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774626.199514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664601728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774626.199531:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774626.199557:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199572:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9f800 x1862032664601088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199586:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664601088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199595:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199598:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199600:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774626.199605:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774626.199622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199630:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9d180 x1862032664601216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199638:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664601216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199646:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774626.199652:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774626.199663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9ce00 x1862032664601344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664601344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199686:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774626.199693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774626.199703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9c700 x1862032664601472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199715:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664601472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774626.199727:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664601600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774626.199736:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9d880 x1862032664601600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199748:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664601600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664601600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774626.199765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664601728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774626.199774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774626.199779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774626/real 1775774626] req@ffff982980f9e300 x1862032664601728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774626.199786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664601728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774626.199791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774626.199793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774626.199794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664601728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774626.711165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774626.967244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774627.735183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774627.991292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774628.759172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774629.015247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774629.783178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774630.039240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774630.807164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774631.063287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774631.319137:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319151:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774631.319156:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319169:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319183:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319191:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774631.319242:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664601856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319277:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774631.319283:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319288:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319292:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319298:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319310:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774631.319323:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664601856:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774631.319329:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664601984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319342:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774631.319348:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319356:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319360:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774631.319376:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664602112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319386:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319391:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774631.319392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319397:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319401:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319405:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774631.319423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664602240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319434:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319437:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774631.319439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774631.319447:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601984:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774631.319447:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319451:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319459:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319466:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774631.319481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664602368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774631.319498:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602112:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774631.319501:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774631.319504:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319520:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319525:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774631.319540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774631.319550:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664602496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774631.319561:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774631.319565:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774631.319567:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774631.319571:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16398 00000100:00080000:3.0:1775774631.319576:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774631.319580:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774631.319585:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774631.319589:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774631.319602:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664602624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774631.319625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664602496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774631.319657:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664602624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774631.831119:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774632.087272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774632.855178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774633.111271:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774633.879195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774634.135287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774634.391346:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774634.391390:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774634.391463:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391502:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391512:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391522:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391535:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391556:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774634.391567:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774634.391573:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9ea00 x1862032664602368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664602368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391626:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391631:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774634.391639:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9e680 x1862032664602496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664602496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664602496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774634.391679:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9df80 x1862032664601984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664601984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664601984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774634.391731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9f100 x1862032664602112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391745:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664602112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774634.391769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9c380 x1862032664602240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664602240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391797:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774634.391846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9fb80 x1862032664601856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391861:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664601856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391869:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664601856:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774634.391918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774631/real 1775774634] req@ffff982980f9d500 x1862032664602624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774634.391932:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664602624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774634.391940:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774634.391944:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774634.391948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664602624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774634.903157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774635.159227:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774635.927192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774636.183244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774636.439178:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439189:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774636.439194:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439205:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439218:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774636.439225:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774636.439255:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664602752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774636.439285:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774636.439291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439296:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439300:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774636.439306:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439316:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774636.439334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664602752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774636.439344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664602880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774636.439357:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774636.439363:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439367:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439372:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774636.439375:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439381:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774636.439394:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664603008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774636.439405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774636.439410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16403 00000800:00000100:3.0:1775774636.439421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774636.439425:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774636.439431:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439438:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774636.439451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664603136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774636.439461:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2d180 x1862032664602752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774636.439470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439480:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774636.439483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439489:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439494:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774636.439497:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664602752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774636.439500:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774636.439516:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.439522:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774636.439526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664603264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774636.439531:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664602752:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774636.439547:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774636.439551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774636.439553:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439558:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439563:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774636.439567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602880:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774636.439573:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439582:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774636.439601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774636.439605:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664603392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774636.439618:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774636.439621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2df80 x1862032664602880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774636.439624:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774636.439629:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774636.439638:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16403 00000100:00080000:0.0:1775774636.439644:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774636.439649:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664602880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774636.439653:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774636.439661:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774636.439664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.439670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.439674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664602880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774636.439679:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664603520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774636.439686:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774636.439717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774636.439732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2d500 x1862032664603008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774636.439745:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664603008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774636.439754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.439757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.439760:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774636.439801:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774636.439825:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774636.439836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2fb80 x1862032664603136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774636.439847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664603136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774636.439868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.439871:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.439875:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774636.439881:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774636.439901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774636.439912:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2f100 x1862032664603264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774636.440029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664603264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774636.440040:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.440044:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.440047:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774636.440097:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664603392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774636.440123:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774636.440136:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2ca80 x1862032664603392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774636.440149:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664603392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774636.440158:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.440162:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.440166:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664603392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774636.440172:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664603520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774636.440194:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774636.440207:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774636/real 1775774636] req@ffff982974b2c380 x1862032664603520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774636.440219:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664603520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774636.440227:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774636.440230:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774636.440234:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664603520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774636.951147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774637.207271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774637.975179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774638.231256:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774638.999139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774639.255202:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774640.023177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774640.279223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774641.047190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774641.303205:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774641.559225:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559233:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774641.559238:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559247:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559256:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559261:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774641.559295:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664603648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774641.559331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774641.559336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559339:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559342:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559346:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774641.559355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774641.559365:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664603776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774641.559369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664603648:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774641.559381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774641.559385:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559388:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559391:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774641.559397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774641.559406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664603904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774641.559413:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559416:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774641.559417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559419:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559421:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559424:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774641.559428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774641.559436:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664604032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774641.559439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774641.559453:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559456:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774641.559468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559471:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559474:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559476:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774641.559480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774641.559485:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9ed80 x1862032664603648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774641.559488:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664604160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774641.559510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774641.559514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774641.559517:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664603648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774641.559520:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559526:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559532:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774641.559537:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775774641.559537:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774641.559545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774641.559549:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664604288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774641.559553:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664603648:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774641.559564:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774641.559567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774641.559568:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774641.559572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16409 00000100:00080000:3.0:1775774641.559574:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774641.559576:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774641.559579:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774641.559582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774641.559587:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664604416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774641.559610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9f480 x1862032664603776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664603776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559641:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559643:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559645:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774641.559649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774641.559659:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9dc00 x1862032664603904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559672:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664603904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559677:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664603904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774641.559717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774641.559744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9d500 x1862032664604032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664604032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774641.559776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774641.559786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9fb80 x1862032664604160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664604160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774641.559810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664604288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774641.559820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9c380 x1862032664604288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664604288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664604288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774641.559854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664604416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774641.559864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774641.559870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774641/real 1775774641] req@ffff982980f9f100 x1862032664604416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774641.559877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664604416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774641.559881:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774641.559883:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774641.559885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664604416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774642.071189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774642.327250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774643.095168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774643.351201:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774644.119159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774644.375229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774645.143172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774645.399260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774646.167161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774646.423324:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774646.679193:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679206:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774646.679213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774646.679225:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679254:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774646.679263:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774646.679316:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664604544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774646.679377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774646.679390:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774646.679398:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679407:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774646.679417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774646.679431:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774646.679456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664604544:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774646.679459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664604672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774646.679485:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774646.679497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774646.679506:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774646.679524:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774646.679535:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774646.679555:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774646.679564:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664604800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774646.679622:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679629:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774646.679633:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774646.679641:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679649:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774646.679653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9df80 x1862032664604544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774646.679660:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774646.679675:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774646.679694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664604544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774646.679701:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664604928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774646.679721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.679737:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774646.679749:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774646.679750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664604544:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774646.679759:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774646.679767:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774646.679774:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679779:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774646.679784:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774646.679792:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774646.679799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774646.679810:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664605056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774646.679829:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679834:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774646.679837:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774646.679844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774646.679851:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679858:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774646.679864:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774646.679870:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774646.679872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9e680 x1862032664604672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774646.679890:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664605184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774646.679900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664604672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774646.679912:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774646.679927:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774646.679933:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774646.679938:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774646.679944:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16414 00000100:00080000:3.0:1775774646.679953:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774646.679954:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774646.679964:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774646.679966:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604672:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775774646.679976:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774646.679984:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774646.679994:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664605312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774646.680017:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774646.680031:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9ea00 x1862032664604800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774646.680047:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664604800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774646.680059:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.680111:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774646.680117:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774646.680125:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774646.680151:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774646.680166:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9e300 x1862032664604928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774646.680180:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664604928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774646.680191:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.680195:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774646.680200:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664604928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774646.680230:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774646.680292:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774646.680316:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9d880 x1862032664605056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774646.680341:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664605056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774646.680354:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.680359:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774646.680363:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774646.680371:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664605184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774646.680400:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774646.680418:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9c700 x1862032664605184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774646.680441:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664605184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774646.680459:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.680465:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774646.680471:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664605184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774646.680482:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664605312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774646.680522:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774646.680545:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774646/real 1775774646] req@ffff982980f9ce00 x1862032664605312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774646.680571:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664605312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774646.680587:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774646.680621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774646.680629:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664605312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774647.191176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774647.447275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774648.215232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774648.471314:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774649.239170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774649.495326:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774650.263206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774650.519305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774651.287196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774651.543247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774651.799318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799327:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774651.799342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799356:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799376:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774651.799387:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774651.799431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664605440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774651.799468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799471:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774651.799474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799480:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799484:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774651.799490:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799512:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774651.799520:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664605440:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774651.799530:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664605568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774651.799543:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799547:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774651.799549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799552:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799556:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774651.799559:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799563:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774651.799572:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664605696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774651.799579:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799582:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774651.799584:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799586:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799590:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774651.799592:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799596:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774651.799606:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664605824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774651.799610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774651.799617:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799621:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774651.799623:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799626:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799630:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774651.799632:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799636:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774651.799644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664605952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774651.799648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2f480 x1862032664605440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774651.799657:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799664:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774651.799665:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799669:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799673:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774651.799673:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664605440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774651.799680:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799687:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774651.799691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774651.799695:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664606080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774651.799697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664605440:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774651.799719:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774651.799722:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774651.799723:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774651.799726:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16419 00000100:00080000:0.0:1775774651.799729:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774651.799732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605568:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774651.799734:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774651.799741:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774651.799750:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664606208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774651.799760:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799770:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2e300 x1862032664605568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664605568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774651.799798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774651.799808:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2e680 x1862032664605696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664605696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774651.799839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774651.799850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2c380 x1862032664605824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664605824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774651.799874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774651.799883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2ca80 x1862032664605952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664605952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799906:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799908:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664605952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774651.799911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774651.799920:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799925:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2f100 x1862032664606080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799931:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664606080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799936:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799940:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774651.799952:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774651.799961:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774651.799967:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774651/real 1775774651] req@ffff982974b2fb80 x1862032664606208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774651.799974:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664606208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774651.799979:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774651.799981:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774651.799983:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774652.311143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774652.567335:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774653.335211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774653.591302:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774654.359233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774654.615320:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774655.383195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774655.639284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774656.407125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774656.663209:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774656.919235:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919246:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774656.919251:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919299:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919312:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919328:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774656.919367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664606336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774656.919397:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774656.919402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919406:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919409:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919443:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774656.919470:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774656.919484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664606464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774656.919497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919502:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775774656.919502:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664606336:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774656.919505:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919517:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919522:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919525:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774656.919530:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774656.919543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664606592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774656.919552:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919555:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774656.919557:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919561:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919564:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919567:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774656.919571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774656.919582:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664606720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774656.919590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774656.919601:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919605:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774656.919608:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919613:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919617:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919620:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774656.919626:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774656.919639:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664606848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774656.919643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe3b80 x1862032664606336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774656.919700:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919705:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774656.919706:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664606336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774656.919707:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919717:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919724:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919730:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774656.919731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775774656.919736:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774656.919739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.919745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664606336:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774656.919750:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664606976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774656.919759:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774656.919762:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774656.919764:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774656.919767:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16424 00000100:00080000:1.0:1775774656.919770:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774656.919773:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774656.919777:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774656.919777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774656.919792:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664607104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774656.919813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.919826:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe1f80 x1862032664606464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.919838:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664606464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.919845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.919848:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.919851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774656.919856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774656.919870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.919884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe0a80 x1862032664606592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.919899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664606592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.919911:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.919914:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.919918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774656.919925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774656.919951:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.919965:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe0e00 x1862032664606720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.919979:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664606720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.919989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.919993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.919997:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774656.920004:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774656.920027:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.920041:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe1c00 x1862032664606848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.920056:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664606848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.920066:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.920070:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.920074:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664606848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774656.920116:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774656.920139:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.920149:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe1500 x1862032664606976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.920159:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664606976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.920166:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.920169:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.920171:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664606976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774656.920188:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664607104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774656.920203:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774656.920210:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774656/real 1775774656] req@ffff982981fe1180 x1862032664607104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774656.920225:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664607104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774656.920235:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774656.920239:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774656.920244:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664607104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774657.431185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774657.687249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774658.455154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774658.711223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774659.479152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774659.735245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774660.503200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774660.759182:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774661.527175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774661.783240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774662.039167:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039176:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774662.039180:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039189:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039199:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039204:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774662.039239:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664607232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039264:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039267:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774662.039268:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039272:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039274:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039277:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774662.039285:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774662.039294:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664607360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039301:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039304:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774662.039306:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039308:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039310:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039315:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039316:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664607232:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775774662.039319:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774662.039344:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664607488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774662.039360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039363:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039365:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039367:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774662.039371:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774662.039378:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664607616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039384:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039387:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774662.039388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039391:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039393:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039395:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774662.039398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774662.039404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664607744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039409:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774662.039413:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774662.039415:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039418:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039420:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774662.039425:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774662.039426:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774662.039440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664607872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774662.039451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774662.039454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774662.039457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774662.039462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe1180 x1862032664607232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774662.039465:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16429 00000100:00080000:1.0:1775774662.039472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774662.039481:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774662.039486:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774662.039492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664607232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774662.039499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664608000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774662.039514:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039522:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039527:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664607232:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774662.039537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774662.039559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe1500 x1862032664607360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664607360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039582:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039585:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774662.039607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774662.039618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe1c00 x1862032664607488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664607488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039638:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039640:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039643:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774662.039646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774662.039657:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe0e00 x1862032664607616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664607616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774662.039681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774662.039690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe0a80 x1862032664607744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664607744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664607744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774662.039714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664607872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774662.039723:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe1f80 x1862032664607872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664607872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664607872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774662.039747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774662.039756:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774662.039761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774662/real 1775774662] req@ffff982981fe3b80 x1862032664608000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774662.039769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664608000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774662.039774:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774662.039775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774662.039778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774662.551194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774662.807260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774663.575196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774663.831274:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774664.599188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774664.855223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774665.623163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774665.879218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774666.647203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774666.903233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774667.159179:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159190:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774667.159195:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159205:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159225:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774667.159279:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664608128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774667.159302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159305:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774667.159306:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159309:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159312:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159314:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774667.159321:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774667.159345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664608256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774667.159353:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159356:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774667.159357:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159360:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159362:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159366:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774667.159370:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664608128:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775774667.159371:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774667.159381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664608384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774667.159388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774667.159391:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159395:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159398:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159400:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774667.159404:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774667.159413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664608512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774667.159419:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774667.159433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159436:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159439:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775774667.159445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775774667.159447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774667.159455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664608640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774667.159462:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774667.159466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159475:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774667.159478:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774667.159487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664608768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774667.159489:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2d500 x1862032664608128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774667.159502:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774667.159508:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774667.159511:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774667.159516:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16434 00000100:00080000:0.0:1775774667.159518:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774667.159521:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774667.159524:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774667.159531:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664608896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774667.159532:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664608128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.159558:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.159564:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.159570:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664608128:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774667.159578:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774667.159604:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.159618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2df80 x1862032664608256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.159632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664608256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.159642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.159645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.159649:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774667.159699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774667.159725:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.159738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2d180 x1862032664608384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.159752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664608384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.159763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.159768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.159772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774667.159778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774667.159801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.159814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2fb80 x1862032664608512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.159828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664608512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.159837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.159841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.159845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774667.159851:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774667.159872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.159883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2f100 x1862032664608640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.159895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664608640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.159905:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.159909:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.159912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664608640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774667.159934:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774667.159992:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.160007:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2ca80 x1862032664608768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.160038:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664608768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.160047:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.160051:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.160054:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774667.160060:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774667.160120:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774667.160138:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774667/real 1775774667] req@ffff982974b2c380 x1862032664608896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774667.160152:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664608896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774667.160160:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774667.160164:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774667.160168:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664608896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774667.671153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774667.927291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774668.695185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774668.951293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774669.719156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774669.975278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774670.743240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774670.999321:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774671.767174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774672.023232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774672.279243:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279251:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774672.279292:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279303:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279339:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279348:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774672.279419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664609024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774672.279477:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279500:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279503:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774672.279527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664609024:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774672.279530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664609152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279554:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279556:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774672.279557:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279562:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279564:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279568:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279573:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774672.279581:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664609280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279588:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279591:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774672.279592:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279594:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279596:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279598:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279602:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774672.279610:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664609408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279615:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279618:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774672.279619:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279621:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279623:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279625:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279627:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774672.279632:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664609536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279638:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279640:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774672.279641:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279643:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279645:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279647:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279649:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774672.279654:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664609664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774672.279659:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774672.279661:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774672.279662:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774672.279664:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16439 00000100:00080000:3.0:1775774672.279666:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774672.279667:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774672.279670:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774672.279676:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664609792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774672.279746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.279780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9d180 x1862032664609024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.279796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664609024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.279803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.279806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.279809:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664609024:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774672.279843:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774672.279869:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.279877:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9f800 x1862032664609152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.279886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664609152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.279891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.279893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.279895:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774672.279899:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774672.279910:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.279915:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9c000 x1862032664609280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.279923:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664609280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.279928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.279930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.279931:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774672.279935:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774672.279945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.279951:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9ce00 x1862032664609408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.279963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664609408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.279968:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.279969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.279971:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774672.279974:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774672.279984:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.279989:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9c700 x1862032664609536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.279996:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664609536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.280001:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.280003:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.280004:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664609536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774672.280007:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664609664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774672.280016:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.280021:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9d880 x1862032664609664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.280028:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664609664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.280037:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.280040:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.280042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664609664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774672.280045:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664609792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774672.280054:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774672.280059:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774672/real 1775774672] req@ffff982980f9e300 x1862032664609792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774672.280092:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664609792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774672.280098:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774672.280100:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774672.280102:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664609792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774672.791170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774673.047282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774673.815173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774674.071229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774674.839163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774675.095270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774675.863162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774676.119250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774676.887169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774677.143260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774677.399199:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774677.399210:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774677.399216:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774677.399228:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399242:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399251:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774677.399300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664609920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774677.399337:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774677.399341:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774677.399343:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774677.399348:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399353:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399364:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774677.399375:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774677.399384:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664609920:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774677.399393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664610048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774677.399406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774677.399410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774677.399412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774677.399416:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399430:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399435:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774677.399441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774677.399442:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774677.399460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664610176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774677.399470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774677.399474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe0380 x1862032664609920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774677.399476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774677.399480:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774677.399490:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399497:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399502:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774677.399506:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664609920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775774677.399510:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774677.399527:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664610304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774677.399531:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.399542:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775774677.399546:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774677.399550:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664609920:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774677.399552:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774677.399556:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774677.399563:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399568:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399572:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774677.399578:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774677.399583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774677.399591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664610432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774677.399604:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774677.399608:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774677.399610:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774677.399616:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774677.399616:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399625:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774677.399630:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774677.399636:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774677.399638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe0000 x1862032664610048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774677.399651:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664610560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774677.399657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664610048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.399688:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775774677.399691:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774677.399695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775774677.399697:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774677.399701:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774677.399703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774677.399711:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16444 00000100:00080000:1.0:1775774677.399716:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774677.399717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610176:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774677.399723:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774677.399730:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774677.399744:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664610688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774677.399753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774677.399773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe1880 x1862032664610176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774677.399789:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664610176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.399799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.399802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774677.399807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774677.399813:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774677.399849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774677.399863:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe2300 x1862032664610304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774677.399876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664610304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.399885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.399889:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774677.399892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774677.399910:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774677.399933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774677.399945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe2680 x1862032664610432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774677.399958:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664610432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.399967:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.399971:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774677.399974:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774677.399980:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664610560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774677.399999:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774677.400011:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe2a00 x1862032664610560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774677.400023:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664610560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.400032:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.400035:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774677.400039:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664610560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774677.400044:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664610688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774677.400118:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774677.400131:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774677/real 1775774677] req@ffff982981fe3480 x1862032664610688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774677.400144:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664610688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774677.400152:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774677.400156:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774677.400159:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664610688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774677.911119:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774678.167210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774678.935164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774679.191267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774679.959172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774680.215202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774680.983181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774681.239249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774682.007185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774682.263211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774682.519197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774682.519213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774682.519224:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519237:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774682.519245:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774682.519287:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664610816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774682.519318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774682.519325:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774682.519337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519342:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774682.519348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774682.519359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774682.519374:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664610816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774682.519378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664610944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774682.519393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774682.519399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774682.519403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519409:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774682.519413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774682.519419:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774682.519433:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664611072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774682.519444:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774682.519450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774682.519455:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774682.519459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519464:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774682.519467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774682.519473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774682.519487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664611200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774682.519498:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775774682.519506:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9ea00 x1862032664610816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774682.519507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774682.519517:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519525:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774682.519530:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519532:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664610816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774682.519538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774682.519542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519546:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519550:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664610816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774682.519555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664611328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774682.519567:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519570:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774682.519572:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774682.519576:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519581:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774682.519585:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610944:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774682.519588:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774682.519595:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774682.519609:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664611456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774682.519615:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519627:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9e680 x1862032664610944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774682.519628:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774682.519636:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774682.519641:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774682.519645:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664610944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774682.519649:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16450 00000100:00080000:3.0:1775774682.519657:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774682.519658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774682.519665:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519672:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664610944:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775774682.519675:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774682.519678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774682.519695:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664611584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774682.519704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519716:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9df80 x1862032664611072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774682.519724:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664611072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774682.519735:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774682.519745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774682.519757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9f100 x1862032664611200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774682.519781:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664611200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774682.519786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774682.519793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774682.519803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9c380 x1862032664611328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774682.519815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664611328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774682.519820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774682.519858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664611456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774682.519870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9fb80 x1862032664611456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774682.519884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664611456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774682.519889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519891:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519893:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664611456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774682.519896:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664611584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774682.519905:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774682.519910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774682/real 1775774682] req@ffff982980f9d500 x1862032664611584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774682.519917:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664611584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774682.519922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774682.519923:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774682.519925:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664611584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774683.031185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774683.287292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774684.055165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774684.311232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774685.079142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774685.335241:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774686.103168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774686.359262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774687.127171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774687.383263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774687.639166:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639182:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774687.639186:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639196:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639207:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774687.639212:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774687.639244:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664611712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774687.639267:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639270:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774687.639272:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639275:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639277:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774687.639280:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774687.639288:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774687.639297:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664611840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774687.639305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639307:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774687.639309:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639311:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639313:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774687.639316:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774687.639318:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774687.639325:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664611712:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774687.639327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664611968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774687.639339:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639341:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774687.639342:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639345:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639347:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774687.639350:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774687.639353:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774687.639363:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664612096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774687.639369:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639373:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774687.639376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639384:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774687.639390:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774687.639390:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774687.639398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774687.639412:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664612224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774687.639422:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9dc00 x1862032664611712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774687.639426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639435:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774687.639439:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639444:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00100000:0.0:1775774687.639448:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664611712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774687.639448:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774687.639457:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774687.639464:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775774687.639465:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774687.639471:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639475:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664611712:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774687.639479:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664612352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774687.639485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611840:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774687.639493:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774687.639496:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774687.639497:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774687.639502:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16455 00000100:00080000:3.0:1775774687.639506:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774687.639510:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774687.639512:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774687.639517:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774687.639523:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9f480 x1862032664611840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774687.639528:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664612480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774687.639550:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664611840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639558:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639563:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774687.639585:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774687.639598:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774687.639606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9ed80 x1862032664611968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774687.639619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664611968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639628:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664611968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774687.639641:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774687.639664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774687.639675:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9d500 x1862032664612096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774687.639683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664612096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774687.639698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774687.639715:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774687.639725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9fb80 x1862032664612224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774687.639737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664612224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639761:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774687.639767:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664612352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774687.639784:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774687.639791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9c380 x1862032664612352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774687.639799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664612352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664612352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774687.639812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664612480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774687.639822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774687.639828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774687/real 1775774687] req@ffff982980f9f100 x1862032664612480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774687.639835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664612480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774687.639839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774687.639841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774687.639843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664612480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774688.151194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774688.407235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774689.175164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774689.431281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774690.199176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774690.455370:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774691.223175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774691.479295:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774692.247192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774692.503319:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774692.759177:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774692.759190:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774692.759195:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759205:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759217:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759225:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774692.759266:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664612608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774692.759300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774692.759304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774692.759306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759311:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759315:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774692.759327:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774692.759346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664612736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774692.759349:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664612608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774692.759381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774692.759384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774692.759386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759392:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759395:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774692.759403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774692.759415:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664612864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774692.759425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774692.759429:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774692.759430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759435:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759440:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775774692.759445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774692.759450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774692.759460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774692.759477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664612992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774692.759489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774692.759497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775774692.759498:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2e680 x1862032664612608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774692.759503:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759516:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759527:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759537:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774692.759548:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664612608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774692.759550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774692.759568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664613120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774692.759572:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774692.759588:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774692.759591:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774692.759593:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774692.759600:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774692.759608:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664612608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774692.759615:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759623:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759629:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774692.759634:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774692.759643:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664613248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774692.759652:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774692.759658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612736:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774692.759659:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774692.759664:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774692.759674:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16460 00000100:00080000:0.0:1775774692.759677:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774692.759678:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774692.759684:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774692.759698:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664613376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774692.759711:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.759729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2e300 x1862032664612736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.759758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664612736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.759768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.759772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.759776:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774692.759794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774692.759822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.759833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2f480 x1862032664612864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.759846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664612864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.759857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.759860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.759864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774692.759871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774692.759890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.759899:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2c380 x1862032664612992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.759911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664612992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.759921:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.759924:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.759928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664612992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774692.759934:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774692.759952:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.759961:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2ca80 x1862032664613120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.759982:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664613120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.759992:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.759995:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.759998:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774692.760017:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664613248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774692.760036:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.760047:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2f100 x1862032664613248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.760059:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664613248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.760092:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.760096:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.760099:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664613248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774692.760106:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664613376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774692.760124:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774692.760135:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774692/real 1775774692] req@ffff982974b2fb80 x1862032664613376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774692.760147:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664613376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774692.760163:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774692.760167:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774692.760170:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664613376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774693.271164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774693.527191:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774694.295156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774694.551276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774695.319152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774695.575220:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774696.343175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774696.599261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774697.367153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774697.623236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774697.879156:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879165:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774697.879170:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879180:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879191:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879198:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774697.879231:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664613504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774697.879268:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879273:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774697.879275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879280:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879284:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879288:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774697.879306:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664613504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774697.879331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664613632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774697.879341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774697.879345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879348:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879352:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879355:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774697.879367:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664613760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774697.879375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774697.879379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879390:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879393:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879400:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774697.879409:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664613888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774697.879414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613632:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774697.879422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774697.879426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879432:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879438:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774697.879446:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664614016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774697.879456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774697.879461:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613760:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774697.879465:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879474:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879480:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774697.879485:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879492:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774697.879507:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774697.879509:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664614144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774697.879524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774697.879527:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774697.879529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774697.879535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16465 00000100:00080000:0.0:1775774697.879541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774697.879546:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614016:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775774697.879547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774697.879557:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774697.879574:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664614272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774697.879591:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664614144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774697.879621:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664614272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774698.391187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774698.647248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774699.415152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774699.671242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774700.439173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774700.695265:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774700.951325:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774700.951365:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774700.951387:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951434:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951446:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951456:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951467:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951478:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774700.951492:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775774700.951508:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2ca80 x1862032664614144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951542:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664614144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951563:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664614144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774700.951579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2fb80 x1862032664613888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664613888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951607:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951612:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774700.951623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2f100 x1862032664614016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664614016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774700.951712:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2df80 x1862032664613632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951725:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664613632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774700.951750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2d500 x1862032664613760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951765:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664613760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664613760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774700.951789:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2d180 x1862032664613504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951804:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664613504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951814:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664613504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774700.951857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774697/real 1775774700] req@ffff982974b2c380 x1862032664614272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774700.951874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664614272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774700.951882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774700.951887:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774700.951891:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664614272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774701.463186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774701.719257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774702.487164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774702.743237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774702.999247:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999260:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774702.999265:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999275:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999287:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999294:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774702.999345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664614400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999384:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999388:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774702.999390:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999396:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999400:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999404:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999415:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774702.999429:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664614528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999440:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999444:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774702.999446:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999450:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999455:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999459:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999465:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774702.999476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664614656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999486:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999489:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774702.999491:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999495:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999499:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999503:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999508:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774702.999518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664614784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999528:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999537:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774702.999539:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999543:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999547:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999550:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999556:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774702.999566:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664614912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999574:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999577:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774702.999579:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999582:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999585:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999589:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999599:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774702.999624:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664615040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774702.999634:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774702.999637:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774702.999639:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774702.999643:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16470 00000100:00080000:1.0:1775774702.999647:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774702.999651:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774702.999656:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774702.999667:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664615168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774702.999779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664614400:192.168.201.118@tcp:250:kworker.0 00000800:00000100:3.0:1775774702.999847:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774702.999881:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774702/real 1775774702] req@ffff982981fe3480 x1862032664614400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774702.999900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664614400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774702.999910:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774702.999913:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774702.999917:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664614400:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774702.999925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774702.999949:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774702.999960:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774702/real 1775774702] req@ffff982981fe2a00 x1862032664614528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000102:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664614528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000111:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000115:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000117:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774703.000123:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774703.000145:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774703.000156:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774703/real 1775774703] req@ffff982981fe2680 x1862032664614656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000172:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664614656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000180:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000183:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000186:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774703.000191:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774703.000212:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774703.000222:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774703/real 1775774703] req@ffff982981fe2300 x1862032664614784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000233:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664614784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000241:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000244:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000247:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774703.000252:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774703.000272:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774703.000282:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774703/real 1775774703] req@ffff982981fe1880 x1862032664614912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000294:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664614912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000309:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000313:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000316:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664614912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774703.000322:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774703.000342:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774703.000353:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774703/real 1775774703] req@ffff982981fe0000 x1862032664615040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000365:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664615040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000374:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000377:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000381:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774703.000387:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774703.000406:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774703.000433:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774703/real 1775774703] req@ffff982981fe0380 x1862032664615168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774703.000445:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664615168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774703.000452:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774703.000455:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774703.000459:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774703.511182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774703.767238:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774704.535184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774704.791263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774705.559174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774705.815264:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774706.583208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774706.839223:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774707.607186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774707.863196:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774708.119149:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119157:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774708.119162:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774708.119172:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119182:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119188:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774708.119224:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664615296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119260:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119264:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774708.119267:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774708.119271:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119277:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119286:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774708.119297:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664615424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774708.119309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774708.119314:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664615296:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774708.119315:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119321:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119328:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774708.119336:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664615552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119342:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119345:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774708.119347:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774708.119350:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119352:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119354:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774708.119364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664615680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774708.119374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774708.119379:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16475 00000800:00000100:0.0:1775774708.119380:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774708.119383:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119386:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119389:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774708.119397:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664615808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774708.119414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774708.119415:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9df80 x1862032664615296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774708.119418:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119423:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119429:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119432:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774708.119437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664615936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774708.119442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774708.119445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774708.119446:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774708.119448:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16475 00000100:00080000:3.0:1775774708.119450:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774708.119451:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774708.119454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774708.119458:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664616064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774708.119481:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664615296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119489:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119492:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119495:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664615296:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774708.119515:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774708.119550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119560:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9e680 x1862032664615424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664615424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119578:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774708.119582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774708.119592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9ea00 x1862032664615552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664615552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119615:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119617:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774708.119625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774708.119643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9e300 x1862032664615680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664615680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774708.119688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615808:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774708.119708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9d880 x1862032664615808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664615808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119735:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119737:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664615808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774708.119744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774708.119758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9c700 x1862032664615936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664615936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119777:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664615936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774708.119785:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774708.119794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774708.119800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774708/real 1775774708] req@ffff982980f9ce00 x1862032664616064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774708.119807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664616064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774708.119812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774708.119813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774708.119815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774708.631187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774708.887207:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774709.655176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774709.911234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774710.679175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774710.935263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774711.703181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774711.959285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774712.727197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774712.983270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774713.239252:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239264:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774713.239269:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239279:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239292:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239300:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774713.239341:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664616192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774713.239374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774713.239381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239385:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774713.239405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774713.239419:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664616320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774713.239448:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664616192:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774713.239460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774713.239464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239477:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239480:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774713.239484:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774713.239496:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664616448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774713.239503:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239505:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774713.239506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239512:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239514:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774713.239518:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774713.239527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664616576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774713.239529:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774713.239543:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774713.239548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239551:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239554:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239556:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774713.239559:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774713.239567:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664616704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774713.239575:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe3b80 x1862032664616192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774713.239581:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239588:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774713.239591:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239599:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:1.0:1775774713.239603:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239605:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774713.239609:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774713.239614:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664616192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774713.239617:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664616832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774713.239646:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774713.239649:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774713.239650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775774713.239650:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774713.239658:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16480 00000100:00080000:0.0:1775774713.239661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775774713.239662:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774713.239668:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664616192:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775774713.239672:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774713.239683:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664616960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774713.239703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774713.239730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239742:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe1f80 x1862032664616320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.239755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664616320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.239763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.239766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.239770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774713.239777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774713.239795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe0a80 x1862032664616448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.239816:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664616448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.239824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.239827:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.239830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774713.239836:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774713.239854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239863:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe0e00 x1862032664616576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.239875:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664616576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.239882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.239885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.239888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774713.239893:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774713.239909:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe1c00 x1862032664616704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.239929:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664616704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.239943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.239946:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.239950:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664616704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774713.239956:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774713.239975:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.239985:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe1500 x1862032664616832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.239997:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664616832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.240005:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.240008:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.240011:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774713.240016:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774713.240036:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774713.240047:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774713/real 1775774713] req@ffff982981fe1180 x1862032664616960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774713.240059:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664616960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774713.240108:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774713.240113:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774713.240117:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664616960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774713.751209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774714.007313:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774714.775138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774715.031210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774715.799192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774716.055231:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774716.823176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774717.079236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774717.847169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774718.103234:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774718.359184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774718.359195:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774718.359201:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359211:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359224:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359231:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774718.359270:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664617088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774718.359301:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774718.359306:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774718.359308:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359313:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359318:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664617088:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775774718.359335:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774718.359367:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664617216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774718.359380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774718.359384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774718.359386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359396:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774718.359406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774718.359418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664617344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774718.359429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775774718.359434:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774718.359435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774718.359439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359444:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359453:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774718.359460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774718.359477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664617472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774718.359477:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2f480 x1862032664617088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774718.359510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774718.359515:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774718.359517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774718.359522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664617088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774718.359525:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359532:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359539:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774718.359544:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775774718.359546:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774718.359549:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359554:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664617088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774718.359563:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664617600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774718.359575:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774718.359580:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774718.359581:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617216:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774718.359584:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359593:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359597:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774718.359602:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774718.359607:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774718.359618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774718.359622:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664617728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774718.359635:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774718.359638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2e300 x1862032664617216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774718.359642:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774718.359647:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774718.359657:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16485 00000100:00080000:0.0:1775774718.359664:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774718.359668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664617216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774718.359672:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774718.359683:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774718.359684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359692:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774718.359698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774718.359702:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664617856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774718.359720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2e680 x1862032664617344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774718.359739:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664617344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774718.359745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774718.359753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774718.359764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359770:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2c380 x1862032664617472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774718.359783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664617472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774718.359791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359794:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359798:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774718.359804:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774718.359824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2d180 x1862032664617600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774718.359843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664617600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774718.359849:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664617600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774718.359867:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664617728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774718.359880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2d500 x1862032664617728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774718.359902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664617728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774718.359911:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664617728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774718.359925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664617856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774718.359945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774718.359953:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774718/real 1775774718] req@ffff982974b2df80 x1862032664617856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774718.359961:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664617856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774718.359967:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774718.359969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774718.359971:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664617856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774718.871168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774719.127221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774719.895174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774720.151252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774720.919231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774721.175206:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774721.943163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774722.199257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774722.967242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774723.223228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774723.479203:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479212:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774723.479217:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479236:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479242:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774723.479289:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664617984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479312:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774723.479316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479319:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479322:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479325:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774723.479343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664618112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479353:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774723.479356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774723.479359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664617984:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774723.479361:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479365:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479371:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774723.479382:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664618240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479388:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774723.479391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479394:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479396:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479399:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774723.479409:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664618368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479417:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774723.479418:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479421:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479423:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479425:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774723.479433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774723.479439:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664618496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774723.479450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479456:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774723.479459:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479462:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774723.479470:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe1180 x1862032664617984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774723.479471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664618624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774723.479501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774723.479503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774723.479505:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774723.479508:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16490 00000100:00080000:1.0:1775774723.479511:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774723.479530:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664617984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774723.479534:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774723.479543:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774723.479547:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479555:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775774723.479556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664618752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774723.479562:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664617984:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774723.479592:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774723.479635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe1500 x1862032664618112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479671:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664618112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479707:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774723.479713:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774723.479731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe1c00 x1862032664618240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664618240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479756:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774723.479762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774723.479773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe0e00 x1862032664618368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664618368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774723.479798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618496:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774723.479807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe0a80 x1862032664618496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664618496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479827:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664618496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774723.479832:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664618624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774723.479841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe1f80 x1862032664618624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664618624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664618624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774723.479865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664618752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774723.479874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774723.479879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774723/real 1775774723] req@ffff982981fe3b80 x1862032664618752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774723.479887:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664618752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774723.479891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774723.479893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774723.479895:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664618752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774723.991205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774724.247330:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774725.015185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774725.271279:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774726.039200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774726.295349:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774727.063264:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774727.319362:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774728.087227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774728.343330:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774728.599243:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599255:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774728.599259:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599270:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599284:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599291:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774728.599339:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664618880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599372:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774728.599373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599377:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599380:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599382:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774728.599393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774728.599412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664619008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599419:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774728.599422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599424:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599454:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774728.599458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774728.599469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664619136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599481:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774728.599483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599490:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599494:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774728.599498:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774728.599536:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664619264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599545:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599548:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774728.599549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599555:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599558:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599564:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774728.599567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664618880:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774728.599569:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774728.599580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664619392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599590:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599594:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774728.599595:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599600:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599604:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599608:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774728.599614:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774728.599629:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664619520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774728.599640:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774728.599645:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774728.599647:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774728.599651:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16496 00000100:00080000:3.0:1775774728.599656:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774728.599661:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774728.599667:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774728.599684:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664619648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774728.599686:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.599757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9c000 x1862032664618880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.599780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664618880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.599792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.599797:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.599801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664618880:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774728.599810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774728.599836:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.599848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9f800 x1862032664619008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.599860:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664619008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.599869:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.599873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.599876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774728.599892:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774728.599907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.599933:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9d180 x1862032664619136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.599944:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664619136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.599952:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.599955:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.599958:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774728.599992:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774728.600042:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.600060:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9ce00 x1862032664619264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.600081:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664619264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.600095:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.600099:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.600102:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774728.600109:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619392:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774728.600234:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.600263:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9c700 x1862032664619392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.600273:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664619392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.600280:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.600283:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.600286:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774728.600291:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664619520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774728.600307:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.600314:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9d880 x1862032664619520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.600323:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664619520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.600329:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.600332:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.600334:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664619520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774728.600339:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664619648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774728.600352:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774728.600359:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774728/real 1775774728] req@ffff982980f9e300 x1862032664619648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774728.600368:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664619648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774728.600374:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774728.600376:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774728.600379:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664619648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774729.111298:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774729.367309:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774730.135174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774730.391238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774731.159217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774731.415312:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774732.183161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774732.439254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774733.207249:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774733.463307:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774733.719229:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774733.719239:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774733.719244:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719253:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:1.0:1775774733.719266:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774733.719272:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774733.719307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664619776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774733.719332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774733.719336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774733.719337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:1.0:1775774733.719344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774733.719346:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774733.719380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664619904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774733.719393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774733.719398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774733.719400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719415:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00100000:0.0:1775774733.719417:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664619776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774733.719424:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774733.719432:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774733.719461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664620032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774733.719478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774733.719483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774733.719485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719491:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:1.0:1775774733.719497:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774733.719502:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774733.719510:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719516:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774733.719540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664620160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774733.719554:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774733.719555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe0380 x1862032664619776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774733.719575:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774733.719583:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719589:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:1.0:1775774733.719594:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774733.719598:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664619776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774733.719604:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719622:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774733.719626:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.719632:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775774733.719636:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664620288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774733.719640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664619776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774733.719663:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774733.719666:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619904:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774733.719669:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774733.719674:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719682:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:1.0:1775774733.719687:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774733.719693:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719701:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774733.719703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.719726:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe0000 x1862032664619904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774733.719728:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664620416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774733.719760:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774733.719768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664619904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774733.719768:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774733.719773:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774733.719784:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16501 00000100:00080000:0.0:1775774733.719791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775774733.719794:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774733.719800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.719816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664619904:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774733.719822:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774733.719833:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774733.719851:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664620544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774733.719862:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774733.719902:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.719916:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe1880 x1862032664620032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774733.719924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664620032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774733.719932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.719935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.719937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774733.719942:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774733.719955:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.719962:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe2300 x1862032664620160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774733.719969:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664620160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774733.719975:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.719976:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.719978:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774733.719982:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774733.719999:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.720010:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe2680 x1862032664620288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774733.720023:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664620288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774733.720032:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.720036:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.720039:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774733.720046:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664620416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774733.720143:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.720153:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe2a00 x1862032664620416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774733.720162:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664620416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774733.720178:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.720181:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.720183:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664620416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774733.720203:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664620544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774733.720218:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774733.720225:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774733/real 1775774733] req@ffff982981fe3480 x1862032664620544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774733.720232:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664620544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774733.720239:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774733.720241:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774733.720243:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664620544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774734.231151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774734.487343:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774735.255207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774735.511294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774736.279138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774736.535215:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774737.303201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774737.559235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774738.327167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774738.583204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774738.839210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839220:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774738.839224:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839235:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839249:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839255:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774738.839300:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664620672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774738.839334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839343:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774738.839351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774738.839361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664620800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774738.839371:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839374:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839377:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839379:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774738.839382:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774738.839388:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664620928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839404:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775774738.839405:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664620672:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774738.839407:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839420:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839424:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774738.839430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774738.839445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664621056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839467:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774738.839469:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839472:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839480:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774738.839481:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774738.839485:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774738.839497:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664621184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839505:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774738.839507:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774738.839509:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839512:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774738.839518:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839522:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9ea00 x1862032664620672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774738.839523:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774738.839534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664621312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774738.839542:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774738.839545:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664620672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774738.839546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774738.839549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774738.839556:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16506 00000100:00080000:3.0:1775774738.839560:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774738.839563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774738.839563:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774738.839570:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774738.839572:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664620672:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774738.839580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664621440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774738.839588:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774738.839611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9e680 x1862032664620800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839639:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664620800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774738.839687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774738.839704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9df80 x1862032664620928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664620928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664620928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774738.839738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774738.839749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839755:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9f100 x1862032664621056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664621056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774738.839776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774738.839787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9c380 x1862032664621184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664621184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774738.839812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664621312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774738.839822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9fb80 x1862032664621312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664621312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664621312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774738.839846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664621440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774738.839856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774738.839862:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774738/real 1775774738] req@ffff982980f9d500 x1862032664621440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774738.839869:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664621440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774738.839874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774738.839876:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774738.839878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664621440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774739.351173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774739.607268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774740.375200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774740.631209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774741.399161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774741.655227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774742.423219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774742.679290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774743.447176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774743.703258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774743.959279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774743.959295:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959305:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959320:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959326:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774743.959372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664621568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774743.959408:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959411:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959414:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959416:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774743.959438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664621696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959449:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774743.959450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959455:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00100000:0.0:1775774743.959457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664621568:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774743.959457:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959464:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959468:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774743.959475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664621824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959481:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774743.959485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959489:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959490:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959493:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959496:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774743.959505:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664621952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774743.959514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959517:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959519:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959522:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959526:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774743.959533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664622080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959539:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774743.959548:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774743.959561:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774743.959563:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959567:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959574:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959576:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774743.959588:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664622208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774743.959595:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774743.959599:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774743.959600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b5f80 x1862032664621568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774743.959601:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774743.959608:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16511 00000100:00080000:2.0:1775774743.959614:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774743.959616:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774743.959619:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774743.959628:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664622336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774743.959637:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664621568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.959648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.959652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.959656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664621568:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774743.959678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774743.959698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.959834:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b6d80 x1862032664621696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.959854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664621696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.959870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.959875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.959880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774743.959901:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774743.959956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.959966:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b4a80 x1862032664621824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.959982:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664621824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.959989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.959992:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.959994:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774743.959999:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774743.960015:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.960023:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b5180 x1862032664621952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.960030:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664621952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.960051:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.960053:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.960055:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664621952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774743.960060:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774743.960126:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.960138:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b5c00 x1862032664622080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.960150:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664622080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.960158:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.960162:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.960165:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774743.960173:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664622208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774743.960194:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.960202:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b4700 x1862032664622208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.960210:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664622208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.960215:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.960218:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.960220:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664622208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774743.960225:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664622336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774743.960239:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774743.960244:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774743/real 1775774743] req@ffff9829764b6a00 x1862032664622336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774743.960251:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664622336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774743.960258:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774743.960261:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774743.960264:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664622336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774744.471198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774744.727300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774745.495151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774745.751225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774746.519169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774746.775257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774747.543166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774747.799204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774748.567201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774748.823242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774749.079198:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079206:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774749.079210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079219:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079265:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079271:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774749.079305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664622464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079331:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774749.079333:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079336:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079339:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079342:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774749.079350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774749.079358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664622592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774749.079369:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079371:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079375:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774749.079379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774749.079384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664622720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079391:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774749.079396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774749.079399:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664622464:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774749.079401:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079410:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774749.079414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774749.079422:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664622848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774749.079433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079436:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079439:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774749.079444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774749.079450:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664622976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774749.079460:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079463:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079465:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079468:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774749.079473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774749.079479:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774749.079487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664623104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774749.079498:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774749.079502:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774749.079503:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774749.079508:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16516 00000100:00080000:3.0:1775774749.079513:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774749.079518:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079521:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9ed80 x1862032664622464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774749.079525:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774749.079544:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664623232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774749.079546:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664622464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079576:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664622464:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774749.079598:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774749.079625:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079637:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9f480 x1862032664622592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079648:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664622592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079654:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079656:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774749.079677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774749.079690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9dc00 x1862032664622720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664622720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774749.079717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774749.079727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079734:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9d500 x1862032664622848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664622848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774749.079753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774749.079764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9fb80 x1862032664622976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664622976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664622976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774749.079787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664623104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774749.079796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9c380 x1862032664623104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664623104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079832:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079834:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664623104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774749.079839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664623232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774749.079853:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774749.079863:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774749/real 1775774749] req@ffff982980f9f100 x1862032664623232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774749.079874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664623232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774749.079882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774749.079885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774749.079888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664623232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774749.591243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774749.847261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774750.615180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774750.871259:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774751.639155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774751.895207:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774752.663187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774752.919233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774753.687159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774753.943251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774754.199196:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199205:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774754.199212:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199222:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199234:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774754.199278:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664623360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774754.199339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199343:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774754.199345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199350:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199353:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199358:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774754.199372:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774754.199393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664623488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774754.199407:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664623360:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774754.199423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199429:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774754.199432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199440:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199464:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199468:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774754.199475:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774754.199490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664623616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774754.199507:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774754.199514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199518:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774754.199520:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199525:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199532:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199536:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774754.199541:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774754.199551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b5500 x1862032664623360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774754.199556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664623744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774754.199589:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199593:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774754.199596:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774754.199597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664623360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774754.199603:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199610:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199626:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774754.199631:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775774754.199632:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774754.199641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775774754.199647:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664623872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774754.199648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664623360:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774754.199666:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199669:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774754.199671:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199675:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199678:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199682:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774754.199686:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774754.199695:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664624000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774754.199703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623488:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774754.199707:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774754.199736:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774754.199738:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774754.199742:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16521 00000100:00080000:2.0:1775774754.199746:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774754.199751:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774754.199768:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774754.199783:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664624128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774754.199797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.199815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b6680 x1862032664623488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.199829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664623488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.199836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.199840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.199842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774754.199848:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774754.199865:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.199873:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b4000 x1862032664623616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.199883:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664623616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.199891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.199894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.199897:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774754.199903:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774754.199917:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.199924:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b4e00 x1862032664623744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.199945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664623744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.199952:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.199954:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.199957:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774754.199979:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774754.199995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.200011:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b7b80 x1862032664623872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.200022:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664623872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.200030:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.200033:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.200036:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664623872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774754.200055:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774754.200071:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.200139:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b7100 x1862032664624000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.200172:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664624000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.200186:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.200191:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.200194:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774754.200201:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774754.200227:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774754.200239:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774754/real 1775774754] req@ffff9829764b5500 x1862032664624128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774754.200251:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664624128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774754.200259:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774754.200262:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774754.200265:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774754.711174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774754.967260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774755.735172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774755.991255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774756.759194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774757.015402:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774757.783190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774758.039254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774758.807185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774759.063331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774759.319156:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319165:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774759.319174:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319184:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319198:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319206:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774759.319244:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664624256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774759.319265:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319268:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774759.319270:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319273:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319276:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319280:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774759.319297:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774759.319304:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664624256:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774759.319311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664624384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774759.319319:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774759.319324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319329:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319333:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774759.319336:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774759.319344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664624512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774759.319350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319353:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774759.319354:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319356:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319358:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319360:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774759.319364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774759.319367:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774759.319374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664624640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774759.319381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774759.319386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319389:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319391:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319394:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774759.319398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774759.319405:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9df80 x1862032664624256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774759.319407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664624768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774759.319436:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774759.319440:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774759.319444:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664624256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774759.319445:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319449:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319455:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774759.319462:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775774759.319462:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774759.319469:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774759.319473:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664624896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774759.319474:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664624256:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774759.319487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774759.319489:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774759.319491:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774759.319495:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16526 00000100:00080000:3.0:1775774759.319497:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774759.319499:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774759.319503:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774759.319504:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774759.319512:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664625024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774759.319536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319550:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9e680 x1862032664624384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319562:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664624384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319571:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319574:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774759.319582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774759.319602:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319612:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9ea00 x1862032664624512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664624512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319677:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774759.319684:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774759.319703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319712:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9e300 x1862032664624640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664624640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319727:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319729:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774759.319732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774759.319742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9d880 x1862032664624768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664624768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319759:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319787:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664624768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774759.319791:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774759.319804:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319811:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9c700 x1862032664624896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664624896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664624896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774759.319842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774759.319852:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774759.319858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774759/real 1775774759] req@ffff982980f9ce00 x1862032664625024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774759.319866:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664625024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774759.319870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774759.319872:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774759.319874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774759.831196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774760.087258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774760.855221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774761.111298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774761.879130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774762.135360:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774762.903152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774763.159258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774763.927242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774764.183274:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774764.439162:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439177:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774764.439183:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439198:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439213:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439222:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774764.439291:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664625152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439327:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439331:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774764.439333:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439337:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439346:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774764.439371:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664625280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774764.439389:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774764.439390:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664625152:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774764.439394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439400:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439405:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774764.439419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664625408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439431:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774764.439433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439436:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439438:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439441:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439446:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774764.439455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664625536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439463:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439466:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774764.439467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439470:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439481:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439484:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774764.439496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664625664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439503:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439513:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774764.439515:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439522:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439526:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774764.439530:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439550:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774764.439555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774764.439566:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664625792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774764.439577:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774764.439580:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774764.439583:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774764.439587:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16531 00000100:00080000:2.0:1775774764.439592:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774764.439597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625408:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774764.439598:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774764.439607:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774764.439620:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664625920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774764.439628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774764.439659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774764.439679:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774764.439699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774764.951226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774765.207345:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774765.975333:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774766.232464:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774766.999135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774767.255234:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775774767.511448:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1775774767.511538:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775774767.511571:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774767.513553:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774767.513570:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774767.513577:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b6a00 x1862032664625152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775774767.513586:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774767.513600:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774767.513616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664625152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775774767.513617:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774767.513631:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00080000:0.0:1775774767.513634:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664625152:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774767.513668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b6d80 x1862032664625792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664625792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513693:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513695:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774767.513699:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b5f80 x1862032664625920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513706:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664625920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664625920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774767.513720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b4700 x1862032664625280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664625280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513735:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774767.513739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b5c00 x1862032664625408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664625408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774767.513769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b5180 x1862032664625536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664625536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774767.513796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774764/real 1775774767] req@ffff9829764b4a80 x1862032664625664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774767.513807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664625664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774767.513812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774767.513814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774767.513816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664625664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775774768.023156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774768.279267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774769.047272:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774769.303277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774769.559220:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559234:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774769.559239:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559261:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559274:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559281:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774769.559482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664626048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774769.559523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559528:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774769.559531:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559537:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559549:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559556:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774769.559569:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774769.559585:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664626176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774769.559596:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559600:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774769.559602:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559608:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559612:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559620:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774769.559621:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664626048:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775774769.559626:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774769.559640:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664626304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774769.559652:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559656:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774769.559658:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559663:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559668:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559673:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774769.559679:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774769.559692:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664626432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774769.559704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774769.559705:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559710:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774769.559712:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559718:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559723:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559728:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774769.559734:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774769.559748:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664626560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774769.559754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe3480 x1862032664626048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774769.559771:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559776:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774769.559778:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559785:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00100000:0.0:1775774769.559786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664626048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774769.559793:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774769.559803:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774769.559805:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775774769.559810:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774769.559814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.559820:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664626048:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774769.559829:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664626688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774769.559832:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626176:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774769.559850:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774769.559854:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774769.559856:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774769.559861:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16537 00000100:00080000:1.0:1775774769.559867:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774769.559872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774769.559877:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774769.559883:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774769.559890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe2a00 x1862032664626176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774769.559900:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664626816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774769.559911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664626176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.559922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.559926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.559930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774769.559958:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774769.559978:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774769.559990:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe2680 x1862032664626304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774769.560002:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664626304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.560010:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.560013:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.560016:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774769.560021:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774769.560041:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774769.560052:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe2300 x1862032664626432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774769.560063:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664626432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.560115:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.560119:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.560123:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774769.560129:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774769.560149:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774769.560162:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe1880 x1862032664626560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774769.560172:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664626560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.560185:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.560188:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.560191:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664626560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774769.560196:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664626688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774769.560215:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774769.560225:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe0000 x1862032664626688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774769.560238:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664626688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.560246:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.560249:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.560252:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664626688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774769.560258:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664626816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774769.560280:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774769.560291:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774769/real 1775774769] req@ffff982981fe0380 x1862032664626816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774769.560303:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664626816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774769.560313:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774769.560317:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774769.560322:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664626816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774770.071183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774770.327307:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774771.095156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774771.351278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774772.119236:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774772.375291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774773.143175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774773.399216:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774774.167200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774774.424432:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774774.679230:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.679241:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774774.679246:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.679297:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.679311:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.681559:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774774.681606:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664626944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774774.682643:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682648:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774774.682650:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682656:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682664:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682667:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774774.682678:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774774.682692:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664627072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774774.682701:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682704:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774774.682705:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682709:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682711:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682717:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774774.682724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664626944:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774774.682725:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774774.682741:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664627200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774774.682751:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682753:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774774.682755:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682760:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682762:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682764:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774774.682768:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774774.682779:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664627328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774774.682786:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682789:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774774.682790:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682792:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682794:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682796:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774774.682798:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774774.682804:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664627456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774774.682810:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682813:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774774.682814:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682817:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682819:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682822:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774774.682825:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774774.682837:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664627584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774774.682841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774774.682853:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774774.682857:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774774.682859:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774774.682866:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16542 00000100:00080000:3.0:1775774774.682872:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774774.682877:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774774.682883:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774774.682889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9d180 x1862032664626944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774774.682899:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664627712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774774.682908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664626944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.682917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.682921:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.682924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664626944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774774.682937:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774774.682956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.682964:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9f800 x1862032664627072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.682972:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664627072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.682977:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.682979:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.682993:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774774.683015:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774774.683028:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.683035:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9c000 x1862032664627200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.683042:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664627200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.683048:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.683050:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.683052:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774774.683056:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774774.683096:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.683109:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9ce00 x1862032664627328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.683122:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664627328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.683131:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.683134:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.683138:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774774.683144:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774774.683166:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.683176:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9c700 x1862032664627456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.683184:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664627456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.683190:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.683192:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.683194:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774774.683198:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664627584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774774.683210:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.683216:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9d880 x1862032664627584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.683223:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664627584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.683228:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.683229:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.683231:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664627584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774774.683235:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664627712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774774.683245:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774774.683250:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774774/real 1775774774] req@ffff982980f9e300 x1862032664627712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774774.683257:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664627712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774774.683263:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774774.683265:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774774.683267:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664627712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774775.191293:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774775.447236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774776.215200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774776.471244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774777.239230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774777.495385:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774778.263337:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774778.519261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774779.287180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774779.543240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774779.799207:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799216:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774779.799220:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799229:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799241:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774779.799247:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774779.799306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664627840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774779.799334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774779.799344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774779.799367:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664627840:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774779.799383:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664627968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799395:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774779.799397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799400:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774779.799405:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799408:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774779.799416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664628096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774779.799426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799432:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774779.799434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774779.799443:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774779.799447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664628224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799466:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774779.799468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799471:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774779.799478:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799481:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774779.799485:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2f100 x1862032664627840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774779.799491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664628352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774779.799517:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774779.799518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799521:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:0.0:1775774779.799523:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774779.799526:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664627840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774779.799529:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799534:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774779.799543:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664628480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774779.799571:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774779.799573:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774779.799574:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774779.799578:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774779.799582:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16547 00000100:00080000:3.0:1775774779.799582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774779.799588:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774779.799590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664627840:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774779.799593:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774779.799599:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774779.799607:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664628608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774779.799633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774779.799656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2fb80 x1862032664627968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664627968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799687:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664627968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774779.799692:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774779.799704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2ca80 x1862032664628096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664628096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799722:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799724:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774779.799730:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774779.799741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2df80 x1862032664628224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664628224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799759:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774779.799766:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774779.799776:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2d500 x1862032664628352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664628352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774779.799800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664628480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774779.799829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2d180 x1862032664628480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664628480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799866:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799869:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664628480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774779.799875:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664628608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774779.799891:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774779.799898:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774779/real 1775774779] req@ffff982974b2c380 x1862032664628608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774779.799906:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664628608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774779.799911:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774779.799913:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774779.799915:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664628608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774780.311173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774780.567232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774781.335203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774781.591266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774782.359169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774782.615259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774783.383168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774783.639268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774784.407203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774784.665547:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774784.919143:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919151:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774784.919199:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919209:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919248:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774784.919282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664628736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774784.919307:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919312:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774784.919314:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774784.919330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664628736:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774784.919334:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919341:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919346:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774784.919356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774784.919372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664628864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774784.919381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774784.919399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919404:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919409:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774784.919426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774784.919447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664628992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774784.919448:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774784.919461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774784.919467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919472:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919476:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919481:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774784.919488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774784.919498:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2e680 x1862032664628736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774784.919500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664629120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774784.919516:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919525:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774784.919527:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919544:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919549:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919555:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919555:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664628736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774784.919563:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774784.919567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919572:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919575:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664628736:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774784.919590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664629248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774784.919598:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919600:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774784.919602:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919604:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919607:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919610:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628864:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775774784.919616:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774784.919626:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664629376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774784.919642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774784.919645:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774784.919649:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774784.919651:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774784.919654:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16552 00000100:00080000:0.0:1775774784.919657:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774784.919659:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774784.919662:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774784.919668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2e300 x1862032664628864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774784.919675:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664629504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774784.919682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664628864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.919701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919704:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774784.919710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774784.919733:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2f480 x1862032664628992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774784.919759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664628992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.919768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664628992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774784.919786:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774784.919814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2c380 x1862032664629120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774784.919844:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664629120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.919852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919855:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774784.919865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774784.919896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2d180 x1862032664629248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774784.919913:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664629248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.919918:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774784.919935:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664629376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774784.919947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919954:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2d500 x1862032664629376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774784.919961:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664629376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.919966:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.919968:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.919970:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664629376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774784.919974:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664629504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774784.919992:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774784.919999:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774784/real 1775774784] req@ffff982974b2df80 x1862032664629504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774784.920007:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664629504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774784.920011:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774784.920014:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774784.920015:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664629504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774785.431190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774785.687286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774786.455170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774786.711248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774787.479177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774787.735254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774788.503156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774788.759231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774789.527198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774789.783252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774790.039204:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039212:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774790.039216:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039225:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039237:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774790.039276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664629632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774790.039300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039303:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774790.039304:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039310:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774790.039330:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664629760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774790.039337:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774790.039342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774790.039362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664629888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774790.039364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664629632:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774790.039373:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774790.039377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039380:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039382:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039393:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774790.039406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664630016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774790.039414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774790.039420:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039424:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039429:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039432:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775774790.039438:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774790.039464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664630144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774790.039475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774790.039479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2ca80 x1862032664629632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774790.039481:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774790.039485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774790.039493:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039499:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774790.039504:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774790.039513:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664629632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774790.039521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664630272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774790.039535:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774790.039536:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774790.039543:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774790.039545:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774790.039548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774790.039552:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16557 00000100:00080000:0.0:1775774790.039556:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774790.039560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664629632:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774790.039562:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774790.039567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774790.039575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774790.039576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664630400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774790.039612:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039627:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2fb80 x1862032664629760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.039656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664629760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.039665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.039669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.039673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774790.039699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774790.039721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2f100 x1862032664629888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.039745:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664629888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.039754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.039758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.039762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664629888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774790.039768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630016:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774790.039790:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2df80 x1862032664630016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.039812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664630016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.039821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.039824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.039828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774790.039834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630144:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774790.039854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039865:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2d500 x1862032664630144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.039877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664630144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.039885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.039888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.039892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774790.039898:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664630272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774790.039918:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039930:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2d180 x1862032664630272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.039941:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664630272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.039950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.039953:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.039956:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664630272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774790.039962:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664630400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774790.039982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774790.039994:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774790/real 1775774790] req@ffff982974b2c380 x1862032664630400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774790.040006:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664630400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774790.040014:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774790.040017:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774790.040020:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664630400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774790.551182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774790.807258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774791.575201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774791.831263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774792.599138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774792.855246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774793.623178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774793.879269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774794.647194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774794.903217:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774795.159194:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.159205:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774795.159210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.159221:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.159241:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.159249:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774795.160376:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664630528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160398:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160402:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774795.160403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160407:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160416:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160425:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774795.160431:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664630528:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774795.160438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664630656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774795.160449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160452:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160454:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160456:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774795.160465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664630784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160471:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774795.160474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160478:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160480:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160482:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160485:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774795.160491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664630912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160498:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774795.160500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160502:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160504:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160506:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774795.160514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664631040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160519:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160521:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774795.160522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160523:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160525:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160527:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774795.160535:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664631168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774795.160540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774795.160543:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774795.160544:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774795.160546:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16562 00000100:00080000:3.0:1775774795.160548:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774795.160549:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774795.160552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774795.160556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664631296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774795.161735:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.161776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9ea00 x1862032664630528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.161801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664630528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.161812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.161816:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.161821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664630528:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774795.161859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774795.161880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.161890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9e680 x1862032664630656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.161899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664630656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.161904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.161907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.161909:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774795.161913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774795.161924:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.161930:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9df80 x1862032664630784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.161937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664630784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.161947:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.161949:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.161951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774795.161955:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774795.161965:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.161971:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9f100 x1862032664630912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.161977:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664630912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.161982:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.161984:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.161986:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664630912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774795.161990:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774795.162018:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.162027:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9c380 x1862032664631040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.162034:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664631040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.162040:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.162042:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.162045:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774795.162049:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664631168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774795.162061:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.162082:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9fb80 x1862032664631168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.162090:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664631168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.162095:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.162097:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.162099:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664631168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774795.162103:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664631296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774795.162114:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774795.162120:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774795/real 1775774795] req@ffff982980f9d500 x1862032664631296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774795.162127:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664631296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774795.162131:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774795.162133:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774795.162135:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664631296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774795.671174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774795.927263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774796.695170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774796.951200:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774797.719133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774797.975254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774798.743190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774798.999216:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774799.767148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774800.023199:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774800.279182:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279190:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774800.279195:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279203:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279213:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279219:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774800.279251:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664631424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774800.279276:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279279:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774800.279281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279286:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279289:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279296:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774800.279306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664631552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774800.279313:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774800.279316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279319:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279321:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774800.279330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664631424:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774800.279345:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279349:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774800.279357:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664631680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774800.279363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279366:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774800.279367:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279370:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279373:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279375:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279378:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774800.279386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664631808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774800.279393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774800.279397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279399:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279401:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279405:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775774800.279409:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774800.279419:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664631936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774800.279425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774800.279429:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279432:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279437:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774800.279449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664632064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774800.279451:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2f480 x1862032664631424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774800.279460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774800.279468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774800.279471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774800.279479:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16567 00000100:00080000:0.0:1775774800.279481:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774800.279484:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774800.279487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774800.279493:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664631424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774800.279496:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664632192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774800.279531:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279538:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279544:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664631424:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774800.279553:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774800.279578:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2e300 x1862032664631552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279606:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664631552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279615:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279619:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279622:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774800.279655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774800.279677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2e680 x1862032664631680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664631680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774800.279724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631808:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774800.279744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2c380 x1862032664631808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664631808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774800.279795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774800.279813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2d180 x1862032664631936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664631936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279849:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279852:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664631936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774800.279858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774800.279877:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279887:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2d500 x1862032664632064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664632064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279925:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.279933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774800.279940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774800.279962:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774800.279973:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774800/real 1775774800] req@ffff982974b2df80 x1862032664632192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774800.279985:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664632192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774800.279994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774800.279998:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774800.280001:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774800.791173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774801.047225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774801.815147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774802.071253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774802.839175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774803.095238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774803.863152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774804.119254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774804.887175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774805.143243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774805.399199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774805.399212:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399221:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399232:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399238:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774805.399276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664632320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399308:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399312:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774805.399315:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399320:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399325:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399339:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774805.399340:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664632320:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774805.399353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664632448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399361:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774805.399365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399367:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399371:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774805.399384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664632576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399393:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774805.399394:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399396:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399399:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399401:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774805.399413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664632704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774805.399428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399431:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399435:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399438:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774805.399443:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774805.399449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664632832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774805.399462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774805.399464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399469:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16572 00000100:00080000:3.0:1775774805.399473:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774805.399474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9dc00 x1862032664632320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774805.399482:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774805.399495:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664632320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399506:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399510:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664632320:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774805.399519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664632960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774805.399532:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774805.399533:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632448:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774805.399542:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774805.399545:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774805.399557:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16572 00000800:00000100:0.0:1775774805.399562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774805.399565:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774805.399571:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774805.399576:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774805.399581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9f480 x1862032664632448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774805.399590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664633088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774805.399606:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664632448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774805.399646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774805.399670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774805.399682:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9ed80 x1862032664632576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774805.399726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664632576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774805.399741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774805.399765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774805.399777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9d500 x1862032664632704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774805.399797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664632704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399806:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399810:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399813:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774805.399835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774805.399856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774805.399868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9fb80 x1862032664632832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774805.399881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664632832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664632832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774805.399924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774805.399945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774805.399957:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9c380 x1862032664632960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774805.399970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664632960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.399979:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.399982:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.399986:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664632960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774805.399992:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774805.400011:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774805.400021:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774805/real 1775774805] req@ffff982980f9f100 x1862032664633088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774805.400033:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664633088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774805.400042:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774805.400046:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774805.400050:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774805.911133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774806.167210:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774806.935246:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774807.191289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774807.959143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774808.215209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774808.983165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774809.239230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774810.007141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774810.263209:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774810.519127:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519134:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774810.519138:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519146:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519157:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519163:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774810.519193:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664633216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519213:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519216:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774810.519218:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519221:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519223:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519225:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774810.519232:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774810.519242:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664633344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519255:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774810.519257:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664633216:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774810.519257:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519264:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519266:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519268:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774810.519271:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774810.519279:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664633472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519285:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519288:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774810.519289:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519291:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519294:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519296:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774810.519299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774810.519305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664633600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519311:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519313:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774810.519314:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519317:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519319:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519324:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774810.519325:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774810.519329:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774810.519337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664633728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774810.519346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774810.519347:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519350:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:3.0:1775774810.519353:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519357:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774810.519361:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774810.519362:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9df80 x1862032664633216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774810.519373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664633856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774810.519384:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774810.519388:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664633216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774810.519390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774810.519393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774810.519402:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16578 00000100:00080000:0.0:1775774810.519409:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774810.519409:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774810.519415:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774810.519420:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775774810.519421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774810.519428:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664633216:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774810.519434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664633984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774810.519462:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774810.519484:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519494:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9e680 x1862032664633344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519509:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664633344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519515:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519520:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774810.519525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774810.519537:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519544:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9ea00 x1862032664633472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519551:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664633472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519556:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519558:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774810.519564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774810.519575:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9e300 x1862032664633600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519588:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664633600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519593:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519595:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774810.519600:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774810.519611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9d880 x1862032664633728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519624:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664633728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664633728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774810.519635:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774810.519645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9c700 x1862032664633856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664633856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774810.519668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774810.519678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774810.519683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774810/real 1775774810] req@ffff982980f9ce00 x1862032664633984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774810.519695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664633984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774810.519700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774810.519702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774810.519704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664633984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774811.031200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774811.287240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774812.055189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774812.311278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774813.079199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774813.335271:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774814.103132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774814.359200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774815.127187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774815.383229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774815.639216:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639226:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774815.639231:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639243:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639256:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774815.639262:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774815.639303:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664634112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774815.639333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774815.639340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639346:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774815.639357:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774815.639370:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774815.639388:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664634240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774815.639404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774815.639413:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639420:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639426:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774815.639433:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774815.639439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774815.639450:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664634112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774815.639455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664634368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774815.639473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774815.639478:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639481:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639486:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774815.639489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774815.639493:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774815.639518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664634496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774815.639532:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639538:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774815.639540:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639546:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639552:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774815.639553:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774815.639563:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774815.639572:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774815.639591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664634624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774815.639605:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b4a80 x1862032664634112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774815.639609:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639620:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774815.639624:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774815.639631:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639637:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774815.639668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664634112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.639682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774815.639686:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774815.639692:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775774815.639697:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774815.639698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664634112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774815.639711:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774815.639727:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664634752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774815.639739:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774815.639743:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774815.639745:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774815.639749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774815.639755:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16583 00000100:00080000:2.0:1775774815.639761:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774815.639765:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774815.639770:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774815.639771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b5180 x1862032664634240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774815.639787:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664634880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774815.639814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664634240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.639823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.639826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.639829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774815.639856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774815.639883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774815.639891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b5c00 x1862032664634368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774815.639902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664634368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.639909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.639911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.639914:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774815.639919:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634496:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774815.639944:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774815.639952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b4700 x1862032664634496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774815.639961:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664634496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.639974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.639976:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.639979:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774815.639983:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634624:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774815.639997:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774815.640004:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b5f80 x1862032664634624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774815.640014:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664634624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.640021:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.640023:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.640025:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664634624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774815.640030:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664634752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774815.640044:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774815.640051:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b6d80 x1862032664634752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774815.640060:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664634752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.640120:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.640123:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.640126:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664634752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774815.640132:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664634880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774815.640149:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774815.640159:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774815/real 1775774815] req@ffff9829764b6a00 x1862032664634880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774815.640169:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664634880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774815.640176:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774815.640179:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774815.640181:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664634880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774816.151227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774816.407311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774817.175160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774817.431258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774818.199245:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774818.455299:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774819.223195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774819.479276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774820.247175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774820.503237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774820.759159:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.759168:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774820.759172:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.759181:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.759192:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.759198:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774820.759229:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664635008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.759252:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.759255:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774820.759257:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.759260:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.759263:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.759266:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.759276:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774820.759288:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664635136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.759297:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.759301:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774820.759303:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.759307:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.759311:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.759315:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.759320:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774820.759326:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664635008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774820.759335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664635264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.759345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.759354:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774820.759356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.759360:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.759364:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.759368:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.759373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774820.759390:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.759419:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2f100 x1862032664635008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.759432:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664635008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.759440:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.759443:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.759445:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664635008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774820.759450:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774820.759464:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.759471:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2fb80 x1862032664635136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.759478:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664635136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.759483:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.759485:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.759487:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774820.759506:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774820.759517:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.759532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2ca80 x1862032664635264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.759546:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664635264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.759575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.759578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.759581:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774820.760500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664635392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.760512:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.760515:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774820.760516:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.760518:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.760521:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.760524:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.760527:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774820.760532:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664635520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.760538:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.760540:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774820.760541:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.760543:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.760544:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.760547:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.760549:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774820.760567:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664635648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774820.760574:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774820.760576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774820.760578:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774820.760582:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16588 00000100:00080000:0.0:1775774820.760585:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774820.760588:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774820.760592:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774820.760595:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774820.760601:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664635776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774820.760627:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.760640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2ca80 x1862032664635392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.760651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664635392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.760658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.760660:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.760663:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774820.760667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774820.760679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.760685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2fb80 x1862032664635520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.760693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664635520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.760702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.760705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.760707:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664635520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774820.760710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664635648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774820.760720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.760725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2f100 x1862032664635648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.760732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664635648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.760737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.760739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.760741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664635648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774820.760754:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664635776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774820.760772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774820.760782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774820/real 1775774820] req@ffff982974b2df80 x1862032664635776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774820.760795:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664635776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774820.760804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774820.760807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774820.760811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664635776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774821.271162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774821.527287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774822.295179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774822.551288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774823.319220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774823.575332:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774824.343231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774824.599276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774825.367166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774825.623244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774825.879197:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774825.879212:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774825.879217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774825.879232:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879245:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879253:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774825.879300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664635904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774825.879333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774825.879336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774825.879338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774825.879343:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879349:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879355:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774825.879373:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664635904:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774825.879386:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664636032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774825.879400:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774825.879404:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774825.879406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774825.879409:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879414:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879419:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879425:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774825.879439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664636160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774825.879448:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774825.879450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774825.879452:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774825.879486:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774825.879496:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879504:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879511:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774825.879526:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664636288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774825.879534:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774825.879537:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774825.879538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774825.879544:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b5500 x1862032664635904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774825.879549:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879556:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879567:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774825.879584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664635904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774825.879590:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664636416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774825.879610:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774825.879619:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774825.879620:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.879626:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664635904:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774825.879633:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774825.879639:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774825.879647:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00080000:2.0:1775774825.879652:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879657:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879665:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774825.879668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774825.879687:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664636544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774825.879702:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775774825.879707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774825.879714:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774825.879718:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774825.879725:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16593 00000100:00100000:3.0:1775774825.879728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b7100 x1862032664636032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774825.879733:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774825.879746:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774825.879754:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774825.879754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664636032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774825.879778:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664636672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774825.879778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.879798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.879805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774825.879815:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774825.879845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774825.879856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b7b80 x1862032664636160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774825.879865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664636160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774825.879872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.879875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.879877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774825.879882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774825.879896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774825.879902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b4e00 x1862032664636288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774825.879910:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664636288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774825.879915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.879918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.879919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774825.879923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774825.879934:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774825.879940:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b4000 x1862032664636416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774825.879954:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664636416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774825.879960:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.879963:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.879965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774825.879984:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664636544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774825.879996:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774825.880002:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b6680 x1862032664636544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774825.880009:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664636544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774825.880015:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.880017:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.880019:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664636544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774825.880023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664636672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774825.880033:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774825.880039:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774825/real 1775774825] req@ffff9829764b5500 x1862032664636672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774825.880046:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664636672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774825.880051:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774825.880053:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774825.880056:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664636672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774826.391209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774826.647291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774827.415187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774827.671298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774828.439181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774828.695266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774829.463168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774829.719262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774830.487173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774830.743300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774830.999162:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999173:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774830.999179:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999204:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999225:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774830.999272:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664636800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774830.999296:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999299:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774830.999301:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999304:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999307:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999310:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774830.999319:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774830.999329:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664636928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774830.999337:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664636800:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774830.999340:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774830.999347:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999355:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774830.999358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774830.999366:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664637056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774830.999372:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999407:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774830.999409:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999421:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999429:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999433:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774830.999439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774830.999451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664637184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774830.999457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636928:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774830.999466:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999469:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774830.999471:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999482:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999486:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999491:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774830.999492:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637056:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775774830.999498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774830.999511:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664637312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774830.999524:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999528:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774830.999530:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774830.999533:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00100000:0.0:1775774830.999536:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637184:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775774830.999538:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999545:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774830.999551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774830.999561:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664637440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774830.999568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774830.999571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774830.999572:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774830.999575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637312:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775774830.999578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16598 00000100:00080000:3.0:1775774830.999582:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774830.999586:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774830.999589:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774830.999598:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664637568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774830.999623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664637440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774830.999643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664637568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774831.511235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774831.767216:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774832.535181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774832.791235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774833.559208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774833.815265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775774834.072490:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775774834.072533:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775774834.072552:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072600:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072609:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072617:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072625:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072634:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775774834.072650:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774834.072663:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9d880 x1862032664637440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664637440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072713:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664637440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774834.072721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9e300 x1862032664637568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664637568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664637568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774834.072758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9ce00 x1862032664637184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664637184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774834.072792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9c700 x1862032664637312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072804:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664637312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774834.072822:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9f800 x1862032664636928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664636928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664636928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774834.072852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9d180 x1862032664637056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072862:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664637056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072869:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774834.072882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774830/real 1775774834] req@ffff982980f9c000 x1862032664636800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774834.072893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664636800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774834.072899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774834.072902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774834.072905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664636800:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774834.583171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774834.839253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774835.607172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774835.863251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774836.119206:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774836.119222:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119232:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119245:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774836.119251:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774836.119309:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664637696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774836.119350:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119354:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774836.119355:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119360:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119366:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774836.119371:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119381:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774836.119402:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664637696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774836.119403:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664637824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774836.119423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774836.119429:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119433:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119438:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774836.119441:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774836.119462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664637952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774836.119473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119477:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774836.119481:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774836.119482:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774836.119490:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119496:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774836.119501:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119508:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774836.119521:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664638080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774836.119524:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2d500 x1862032664637696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774836.119560:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119565:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775774836.119566:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664637696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774836.119570:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119579:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119586:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774836.119593:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119601:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774836.119601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.119609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.119615:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664637696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774836.119618:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664638208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774836.119631:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774836.119635:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774836.119636:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119640:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119644:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774836.119648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637824:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774836.119665:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119671:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774836.119682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774836.119686:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664638336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774836.119699:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774836.119702:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2d180 x1862032664637824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774836.119704:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774836.119708:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774836.119718:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16603 00000100:00080000:0.0:1775774836.119725:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774836.119727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664637824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774836.119732:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774836.119742:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774836.119745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.119752:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.119757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774836.119762:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664638464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774836.119770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774836.119807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774836.119819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2c380 x1862032664637952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774836.119830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664637952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774836.119840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.119843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.119847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664637952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774836.119853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774836.119876:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774836.119891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2e680 x1862032664638080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774836.119903:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664638080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774836.119911:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.119914:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.119917:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774836.119923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774836.120041:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774836.120055:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2e300 x1862032664638208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774836.120091:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664638208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774836.120100:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.120104:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.120107:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774836.120135:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664638336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774836.120159:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774836.120171:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2f480 x1862032664638336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774836.120184:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664638336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774836.120192:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.120195:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.120199:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664638336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774836.120205:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664638464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774836.120225:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774836.120236:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774836/real 1775774836] req@ffff982974b2d500 x1862032664638464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774836.120248:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664638464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774836.120256:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774836.120260:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774836.120263:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664638464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774836.631211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774836.887329:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774837.655168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774837.911228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774838.679181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774838.935276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774839.703257:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774839.959342:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774840.727199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774840.983236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774841.239219:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774841.239231:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239240:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239257:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239263:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774841.239296:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664638592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239324:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239327:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774841.239329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239337:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239346:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774841.239354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664638720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239361:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774841.239365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239368:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239371:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239373:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239376:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774841.239382:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664638592:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774841.239385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664638848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774841.239400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239403:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239404:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239407:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239411:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774841.239417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664638976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239426:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774841.239428:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239431:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239442:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774841.239454:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774841.239456:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664639104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239463:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774841.239466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774841.239469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239471:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239474:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774841.239482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664639232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774841.239489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774841.239491:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774841.239493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774841.239496:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b6a00 x1862032664638592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774841.239497:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16608 00000100:00080000:2.0:1775774841.239502:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774841.239506:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774841.239511:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774841.239523:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664639360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774841.239531:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664638592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239551:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664638592:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774841.239565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774841.239592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b6d80 x1862032664638720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664638720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239632:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774841.239662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774841.239684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b5f80 x1862032664638848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664638848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239721:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774841.239732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774841.239753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b4700 x1862032664638976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664638976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664638976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774841.239799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774841.239819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b5c00 x1862032664639104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239842:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664639104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239850:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239857:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774841.239863:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664639232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774841.239883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b5180 x1862032664639232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664639232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664639232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774841.239928:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664639360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774841.239947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774841.239958:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774841/real 1775774841] req@ffff9829764b4a80 x1862032664639360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774841.239970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664639360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774841.239978:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774841.239981:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774841.239985:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664639360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774841.751227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774842.007313:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774842.775197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774843.031300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774843.799211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774844.055230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774844.823208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774845.079269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774845.847218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774846.103267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774846.359228:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359239:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774846.359244:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359254:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359267:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774846.359274:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774846.359314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664639488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774846.359341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774846.359345:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359349:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359352:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774846.359355:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774846.359364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774846.359374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664639616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774846.359383:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774846.359387:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00100000:0.0:1775774846.359393:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664639488:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774846.359394:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774846.359401:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774846.359405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774846.359412:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664639744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774846.359418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774846.359423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359429:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774846.359431:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774846.359435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774846.359442:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664639872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774846.359448:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774846.359451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359454:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359456:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774846.359461:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774846.359461:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774846.359474:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774846.359481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664640000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774846.359487:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359490:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774846.359492:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774846.359496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359498:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774846.359502:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359503:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9ea00 x1862032664639488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774846.359506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774846.359518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664640128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774846.359525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664639488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774846.359527:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774846.359536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774846.359540:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774846.359546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774846.359546:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16613 00000100:00080000:3.0:1775774846.359551:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774846.359554:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774846.359558:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359561:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664639488:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774846.359563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774846.359573:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664640256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774846.359586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774846.359607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9e680 x1862032664639616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664639616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359635:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774846.359639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774846.359650:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9df80 x1862032664639744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359662:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664639744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774846.359675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774846.359695:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9f100 x1862032664639872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359734:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664639872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664639872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774846.359749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774846.359761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9c380 x1862032664640000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664640000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774846.359788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664640128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774846.359798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9fb80 x1862032664640128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664640128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359822:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359824:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664640128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774846.359827:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664640256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774846.359836:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774846.359842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774846/real 1775774846] req@ffff982980f9d500 x1862032664640256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774846.359848:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664640256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774846.359853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774846.359856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774846.359859:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664640256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774846.871193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774847.127226:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774847.895212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774848.151259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774848.919179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774849.175269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774849.943184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774850.199280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774850.967169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774851.223256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774851.479159:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479169:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774851.479174:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479184:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479197:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774851.479203:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774851.479267:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664640384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774851.479291:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479294:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774851.479295:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479299:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479302:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774851.479305:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774851.479316:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774851.479329:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664640512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774851.479340:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774851.479346:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479351:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479355:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774851.479362:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664640384:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774851.479368:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774851.479381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664640640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774851.479389:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479392:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774851.479394:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479396:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479400:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774851.479403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774851.479407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774851.479416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664640768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774851.479422:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774851.479427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479429:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479432:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774851.479435:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774851.479439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774851.479440:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774851.479448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664640896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774851.479458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479462:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774851.479464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774851.479469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774851.479478:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9ed80 x1862032664640384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774851.479482:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774851.479492:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774851.479500:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664640384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774851.479508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664641024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774851.479526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774851.479528:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774851.479532:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774851.479533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774851.479535:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774851.479540:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664640384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774851.479541:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16618 00000100:00080000:3.0:1775774851.479548:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774851.479551:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640512:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774851.479554:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774851.479572:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774851.479583:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664641152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774851.479592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479603:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9f480 x1862032664640512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664640512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479621:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774851.479644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774851.479655:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479661:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9dc00 x1862032664640640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664640640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774851.479682:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774851.479692:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479698:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9d500 x1862032664640768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664640768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774851.479717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774851.479727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9fb80 x1862032664640896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664640896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479746:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664640896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774851.479751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774851.479760:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9c380 x1862032664641024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479795:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664641024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774851.479808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774851.479819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774851.479835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774851/real 1775774851] req@ffff982980f9f100 x1862032664641152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774851.479842:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664641152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774851.479847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774851.479849:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774851.479851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774851.991184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774852.247246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774853.015137:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774853.271282:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774854.039199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774854.295243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774855.063170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774855.319214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774856.087200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774856.343238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774856.599203:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599211:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774856.599217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599226:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599237:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774856.599276:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664641280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774856.599298:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599301:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774856.599302:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599305:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599307:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599310:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599318:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774856.599332:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664641408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774856.599341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774856.599342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664641280:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774856.599346:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774856.599349:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599354:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599356:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599358:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599361:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774856.599370:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664641536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774856.599381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774856.599385:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599388:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599390:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599392:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599396:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774856.599404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664641664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774856.599408:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774856.599416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774856.599421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599425:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599430:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599434:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774856.599441:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664641792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774856.599445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2df80 x1862032664641280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774856.599455:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599463:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774856.599465:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00100000:3.0:1775774856.599473:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664641280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774856.599474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599482:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599489:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774856.599491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599496:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774856.599499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664641920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774856.599503:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664641280:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774856.599514:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774856.599516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774856.599517:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774856.599521:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16624 00000100:00080000:0.0:1775774856.599523:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774856.599526:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774856.599529:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774856.599532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774856.599539:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664642048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774856.599554:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599564:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2f100 x1862032664641408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599573:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664641408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599579:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599584:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774856.599587:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774856.599600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2fb80 x1862032664641536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599614:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664641536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599622:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599624:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774856.599627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641664:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774856.599638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2ca80 x1862032664641664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664641664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599656:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599658:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774856.599667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641792:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774856.599685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2d500 x1862032664641792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664641792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664641792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774856.599732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774856.599746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2f480 x1862032664641920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664641920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664641920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774856.599782:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774856.599792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774856.599798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774856/real 1775774856] req@ffff982974b2e300 x1862032664642048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774856.599805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664642048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774856.599813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774856.599816:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774856.599818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774857.111173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774857.367209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774858.135172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774858.391234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774859.159176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774859.415213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774860.183148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774860.439191:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774861.207166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774861.463242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774861.719263:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719271:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774861.719276:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719285:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719298:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719305:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774861.719350:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664642176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774861.719376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719379:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774861.719380:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719384:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719387:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719390:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774861.719399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774861.719410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664642304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774861.719424:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774861.719429:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664642176:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774861.719429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774861.719434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719445:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719450:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719454:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774861.719461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774861.719475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664642432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774861.719487:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719490:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774861.719492:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719495:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719504:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774861.719510:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774861.719515:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774861.719524:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664642560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774861.719533:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774861.719538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719541:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719544:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719547:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774861.719551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774861.719556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9df80 x1862032664642176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774861.719563:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664642688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774861.719588:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719592:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775774861.719597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664642176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774861.719606:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719616:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:0.0:1775774861.719619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774861.719622:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774861.719628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774861.719631:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774861.719635:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664642176:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774861.719641:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774861.719654:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664642816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774861.719663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642304:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774861.719665:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774861.719669:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774861.719670:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774861.719674:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16629 00000100:00080000:3.0:1775774861.719676:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774861.719683:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774861.719686:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774861.719696:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664642944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774861.719709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.719725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9e680 x1862032664642304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.719735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664642304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.719742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.719744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.719747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774861.719752:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774861.719765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.719772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9ea00 x1862032664642432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.719780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664642432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.719786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.719788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.719791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774861.719796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774861.719810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.719820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9c000 x1862032664642560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.719833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664642560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.719874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.719878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.719882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774861.719889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774861.719910:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.719919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9d180 x1862032664642688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.719929:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664642688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.719935:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.719937:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.719939:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664642688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774861.719953:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774861.719965:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.720014:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9f800 x1862032664642816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.720023:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664642816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.720029:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.720031:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.720033:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774861.720039:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774861.720059:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774861.720098:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774861/real 1775774861] req@ffff982980f9c700 x1862032664642944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774861.720114:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664642944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774861.720120:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774861.720123:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774861.720125:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664642944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774862.231146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774862.487239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774863.255200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774863.511236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774864.279196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774864.535262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774865.303175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774865.559257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774866.327157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774866.583228:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774866.839163:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839173:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774866.839178:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839203:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774866.839247:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664643072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839271:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774866.839275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839284:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839292:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774866.839300:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664643200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839307:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839309:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774866.839310:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839316:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839319:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774866.839324:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664643328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774866.839338:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839341:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839342:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839345:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774866.839354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664643456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839361:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839364:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774866.839365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774866.839366:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664643072:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774866.839369:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839373:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839377:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839385:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774866.839393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664643584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839399:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839402:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774866.839403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839405:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839407:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839410:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839413:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774866.839420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664643712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774866.839425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774866.839428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774866.839429:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774866.839432:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16634 00000100:00080000:3.0:1775774866.839436:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774866.839438:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774866.839441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774866.839449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664643840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774866.839450:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9ce00 x1862032664643072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839511:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664643072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839521:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839525:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839539:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664643072:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774866.839546:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774866.839569:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9e300 x1862032664643200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839593:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664643200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839608:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774866.839614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774866.839631:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9d880 x1862032664643328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664643328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774866.839707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774866.839727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9c700 x1862032664643456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664643456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774866.839801:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774866.839820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9f800 x1862032664643584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664643584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664643584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774866.839868:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664643712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774866.839883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839892:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9d180 x1862032664643712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839903:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664643712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839911:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839914:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664643712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774866.839923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664643840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774866.839940:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774866.839949:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774866/real 1775774866] req@ffff982980f9c000 x1862032664643840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774866.839960:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664643840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774866.839968:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774866.839971:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774866.839975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664643840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774867.351192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774867.607292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774868.375186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774868.631315:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774869.399152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774869.655264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774870.423158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774870.679223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774871.447245:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774871.703291:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774871.959184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959193:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774871.959197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959216:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959222:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774871.959255:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664643968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774871.959278:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774871.959282:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959285:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959287:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959289:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774871.959297:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774871.959304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664644096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774871.959316:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774871.959320:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959323:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959325:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959327:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774871.959330:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774871.959336:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664644224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774871.959343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664643968:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774871.959346:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959351:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774871.959353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959356:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959357:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959360:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774871.959366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774871.959379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664644352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774871.959389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959393:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774871.959395:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959398:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16639 00000800:00000100:0.0:1775774871.959403:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774871.959405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959411:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774871.959417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774871.959429:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664644480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774871.959440:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe3b80 x1862032664643968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774871.959440:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774871.959451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959462:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959468:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774871.959473:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959476:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664643968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775774871.959480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774871.959491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959495:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775774871.959500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664644608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774871.959501:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664643968:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774871.959513:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644096:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774871.959520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774871.959524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774871.959526:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774871.959532:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16639 00000100:00080000:1.0:1775774871.959536:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775774871.959540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774871.959543:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774871.959548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774871.959556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe1f80 x1862032664644096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774871.959561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664644736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774871.959587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664644096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959600:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774871.959620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774871.959637:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959644:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe0a80 x1862032664644224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774871.959652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664644224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959677:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774871.959682:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774871.959697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe0e00 x1862032664644352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774871.959723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664644352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774871.959738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774871.959748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe1c00 x1862032664644480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774871.959760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664644480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774871.959773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664644608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774871.959783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959788:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe1500 x1862032664644608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774871.959796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664644608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664644608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774871.959807:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664644736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774871.959816:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774871.959822:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774871/real 1775774871] req@ffff982981fe1180 x1862032664644736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774871.959829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664644736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774871.959834:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774871.959836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774871.959838:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664644736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774872.471167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774872.727229:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774873.495182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774873.751246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774874.519183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774874.775239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774875.543172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774875.799278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774876.567295:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774876.823257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774877.079177:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079188:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774877.079193:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079204:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079217:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079224:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774877.079263:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664644864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774877.079286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774877.079291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079294:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079308:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079312:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774877.079320:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774877.079331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664644992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774877.079338:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079341:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774877.079342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079353:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664644864:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775774877.079355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774877.079377:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664645120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774877.079388:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079391:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774877.079393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079397:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079402:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774877.079414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774877.079429:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774877.079430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664645248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774877.079447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079450:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774877.079452:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079456:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079460:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079462:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774877.079467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774877.079478:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664645376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774877.079479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9ea00 x1862032664644864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774877.079491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774877.079498:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774877.079501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079507:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079510:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079513:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774877.079516:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774877.079519:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664644864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774877.079526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664645504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774877.079532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774877.079535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774877.079539:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774877.079540:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774877.079542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774877.079545:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16644 00000100:00080000:3.0:1775774877.079548:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774877.079550:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774877.079552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774877.079556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664645632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774877.079568:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664644864:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774877.079589:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774877.079611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9e680 x1862032664644992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664644992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079657:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079659:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664644992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774877.079681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774877.079694:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9df80 x1862032664645120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664645120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774877.079721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774877.079732:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9f100 x1862032664645248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664645248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079752:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774877.079758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774877.079768:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9c380 x1862032664645376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664645376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774877.079792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664645504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774877.079801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079809:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9fb80 x1862032664645504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664645504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664645504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774877.079842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664645632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774877.079862:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774877.079872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774877/real 1775774877] req@ffff982980f9d500 x1862032664645632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774877.079880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664645632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774877.079886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774877.079888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774877.079889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664645632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774877.591176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774877.847233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774878.615177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774878.871285:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774879.639140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774879.895221:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774880.663203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774880.919212:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775774881.687191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774881.943206:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774882.199182:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199189:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774882.199194:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199203:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199214:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199220:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774882.199254:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664645760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774882.199284:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199287:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774882.199288:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199291:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199294:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199297:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774882.199305:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774882.199314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664645888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774882.199321:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774882.199325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199329:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199337:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664645760:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775774882.199339:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774882.199350:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664646016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774882.199357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199361:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774882.199363:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199367:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199371:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199376:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774882.199383:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774882.199396:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664646144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774882.199407:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774882.199408:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774882.199416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199421:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199425:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199429:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774882.199433:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774882.199443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664646272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774882.199452:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b4a80 x1862032664645760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774882.199453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774882.199461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774882.199464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774882.199471:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774882.199494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774882.199496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664645760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199508:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775774882.199508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664646400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774882.199518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774882.199523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774882.199524:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664645760:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774882.199531:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774882.199534:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774882.199537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645888:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774882.199538:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16649 00000100:00080000:2.0:1775774882.199545:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774882.199547:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774882.199551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774882.199562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664646528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774882.199570:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b5180 x1862032664645888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199596:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664645888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199607:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664645888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774882.199625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774882.199637:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199644:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b5c00 x1862032664646016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664646016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774882.199670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646144:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774882.199682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b4700 x1862032664646144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664646144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774882.199707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646272:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774882.199722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b5f80 x1862032664646272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664646272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774882.199770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664646400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774882.199791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b6d80 x1862032664646400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664646400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664646400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774882.199833:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664646528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774882.199849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774882.199857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774882/real 1775774882] req@ffff9829764b6a00 x1862032664646528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774882.199865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664646528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774882.199870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774882.199872:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774882.199885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664646528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774882.711178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774882.967249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774883.735154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774883.991382:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774884.759179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774885.015235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774885.783171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774886.039275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774886.807174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774887.063239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774887.319248:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319256:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774887.319261:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319271:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319282:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319289:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774887.319323:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664646656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319345:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319348:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774887.319349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319355:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319357:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774887.319373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664646784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774887.319382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319385:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319386:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319388:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774887.319415:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664646912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774887.319427:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319432:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319435:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319438:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774887.319445:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664646656:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774887.319448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664647040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319458:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774887.319462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319465:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319468:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319470:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319472:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774887.319481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664647168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774887.319489:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774887.319490:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319492:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319494:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319496:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319499:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774887.319507:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664647296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774887.319513:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774887.319514:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774887.319518:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774887.319520:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774887.319524:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16654 00000100:00080000:3.0:1775774887.319528:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774887.319532:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774887.319537:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774887.319548:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9dc00 x1862032664646656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774887.319551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664647424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774887.319573:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664646656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319581:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319585:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664646656:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774887.319594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774887.319608:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9f480 x1862032664646784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319624:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664646784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774887.319638:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774887.319648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9ed80 x1862032664646912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319660:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664646912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319669:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664646912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774887.319689:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774887.319701:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9d500 x1862032664647040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319714:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664647040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319721:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774887.319726:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774887.319735:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9fb80 x1862032664647168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319748:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664647168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774887.319760:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664647296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774887.319769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9c380 x1862032664647296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319781:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664647296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664647296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774887.319793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664647424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774887.319802:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774887.319807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774887/real 1775774887] req@ffff982980f9f100 x1862032664647424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774887.319814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664647424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774887.319819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774887.319821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774887.319823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664647424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774887.831127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774888.087225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774888.855170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774889.111253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774889.879126:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774890.135220:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774890.903170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774891.159235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774891.927148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774892.183237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774892.439155:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439166:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774892.439171:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439182:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:0.0:1775774892.439209:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774892.439217:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774892.439262:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664647552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774892.439286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439289:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774892.439291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439294:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:0.0:1775774892.439297:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774892.439311:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774892.439320:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774892.439331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664647552:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774892.439340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664647680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774892.439349:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439352:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774892.439353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439356:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:0.0:1775774892.439359:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774892.439362:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774892.439365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774892.439374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664647808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774892.439381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774892.439391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439395:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16659 00000800:00000100:3.0:1775774892.439401:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774892.439404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774892.439411:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774892.439418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774892.439432:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2e680 x1862032664647552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774892.439436:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664647936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774892.439462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774892.439467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774892.439469:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664647552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774892.439473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:0.0:1775774892.439480:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774892.439485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774892.439488:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775774892.439489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774892.439499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774892.439504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664648064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774892.439508:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664647552:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774892.439525:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774892.439529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774892.439531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439536:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:0.0:1775774892.439540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774892.439540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774892.439550:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774892.439556:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774892.439568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2c380 x1862032664647680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774892.439582:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664648192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774892.439608:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774892.439609:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664647680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774892.439614:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774892.439617:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774892.439624:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16659 00000100:00080000:3.0:1775774892.439625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774892.439631:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774892.439631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774892.439641:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647808:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775774892.439651:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774892.439668:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664648320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774892.439673:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2d180 x1862032664647808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774892.439715:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664647808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774892.439722:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774892.439724:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774892.439731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774892.439748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439755:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2e300 x1862032664647936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774892.439763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664647936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774892.439768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774892.439770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664647936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774892.439775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774892.439785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439790:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2f480 x1862032664648064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774892.439797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664648064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774892.439802:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774892.439805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774892.439819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664648192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774892.439829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439834:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2d500 x1862032664648192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774892.439841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664648192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774892.439847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774892.439848:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664648192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774892.439853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664648320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774892.439863:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774892.439868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774892/real 1775774892] req@ffff982974b2ca80 x1862032664648320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774892.439875:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664648320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774892.439891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774892.439893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774892.439896:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664648320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774892.951148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774893.207259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774893.975173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774894.231226:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774894.999167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774895.255240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774896.023177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774896.279277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774897.047186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774897.303344:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774897.559187:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774897.559204:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559228:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559245:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559254:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774897.559304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664648448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774897.559339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559348:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774897.559362:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774897.559371:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664648448:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774897.559375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664648576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774897.559400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559404:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559406:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559411:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774897.559414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774897.559427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664648704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774897.559500:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774897.559537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648704:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774897.559570:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774897.559574:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559582:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559590:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559597:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774897.559607:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774897.559637:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664648832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559663:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559667:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774897.559669:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559674:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559679:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559684:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774897.559687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648832:192.168.201.118@tcp:8:kworker.0 00000008:00100000:2.0:1775774897.559692:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774897.559722:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664648960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559744:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559748:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774897.559750:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774897.559756:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559761:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774897.559762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648960:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775774897.559768:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774897.559778:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774897.559794:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664649088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774897.559810:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774897.559815:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774897.559817:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774897.559822:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649088:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775774897.559824:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16665 00000100:00080000:2.0:1775774897.559832:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774897.559838:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774897.559844:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774897.559861:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664649216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774897.559895:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774898.071197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774898.327307:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774899.095176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774899.351290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774900.119259:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774900.375240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775774900.631436:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1775774900.631487:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775774900.631509:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631577:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631587:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631594:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631602:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631610:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775774900.631618:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775774900.631659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b6a00 x1862032664649216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664649216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631713:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774900.631720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b7100 x1862032664649088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664649088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774900.631756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b7b80 x1862032664648960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664648960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774900.631800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b4e00 x1862032664648832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664648832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774900.631835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b6680 x1862032664648576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664648576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631857:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774900.631867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b4000 x1862032664648704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664648704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631896:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664648704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774900.631903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774897/real 1775774900] req@ffff9829764b5500 x1862032664648448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774900.631916:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664648448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774900.631924:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774900.631928:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774900.631932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664648448:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774901.143166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774901.399214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774902.167185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774902.423203:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774902.679175:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774902.679182:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774902.679186:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679195:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16670 00000100:00080000:2.0:1775774902.679207:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679214:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774902.679254:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664649344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774902.679283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774902.679287:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774902.679296:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679303:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16670 00000100:00080000:2.0:1775774902.679307:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774902.679323:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774902.679335:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664649344:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774902.679340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664649472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774902.679353:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774902.679356:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774902.679358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679362:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16670 00000100:00080000:2.0:1775774902.679366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679369:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774902.679374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774902.679388:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664649600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774902.679398:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774902.679401:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774902.679403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679414:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16670 00000800:00000100:0.0:1775774902.679419:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774902.679421:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679427:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774902.679434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774902.679445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664649728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774902.679456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774902.679459:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b6d80 x1862032664649344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774902.679461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774902.679464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16670 00000100:00080000:2.0:1775774902.679479:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679489:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664649344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774902.679489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774902.679503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775774902.679504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664649856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774902.679513:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774902.679520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774902.679523:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664649344:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774902.679525:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774902.679529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16670 00000100:00080000:2.0:1775774902.679539:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774902.679549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774902.679563:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774902.679567:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664649984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774902.679580:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774902.679584:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774902.679586:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774902.679591:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16670 00000800:00000100:0.0:1775774902.679597:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774902.679597:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774902.679605:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774902.679611:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774902.679620:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b5f80 x1862032664649472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774902.679625:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664650112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774902.679651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664649472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.679662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.679667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.679671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774902.679680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774902.679704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b4700 x1862032664649600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774902.679729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664649600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.679739:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.679743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.679745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774902.679750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774902.679770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679784:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b5c00 x1862032664649728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774902.679796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664649728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.679804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.679807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.679810:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774902.679815:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774902.679836:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b5180 x1862032664649856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774902.679859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664649856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.679867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.679871:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.679874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664649856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774902.679894:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774902.679917:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679928:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b4a80 x1862032664649984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774902.679939:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664649984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.679948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.679951:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.679954:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664649984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774902.679960:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664650112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774902.679978:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774902.679989:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774902/real 1775774902] req@ffff9829764b6d80 x1862032664650112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774902.680010:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664650112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774902.680018:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774902.680021:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774902.680024:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664650112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774903.191178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774903.447218:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774904.215206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774904.471312:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774905.239169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774905.495238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774906.263160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774906.519327:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774907.287147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774907.543230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774907.799185:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799193:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774907.799197:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799206:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799219:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774907.799226:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774907.799267:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664650240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799292:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799296:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774907.799297:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799300:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774907.799307:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774907.799315:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774907.799323:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664650368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799334:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774907.799335:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799340:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799341:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774907.799341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664650240:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774907.799348:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774907.799351:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774907.799360:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664650496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799366:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774907.799370:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799374:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799376:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774907.799379:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774907.799383:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774907.799390:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664650624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774907.799400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799402:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799405:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775774907.799409:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774907.799411:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774907.799415:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774907.799423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664650752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799429:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799431:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774907.799433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799437:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799439:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774907.799444:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799444:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2fb80 x1862032664650240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774907.799447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774907.799458:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664650880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774907.799478:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774907.799480:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774907.799481:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774907.799484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16675 00000100:00080000:0.0:1775774907.799486:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774907.799489:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664650240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774907.799491:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774907.799496:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774907.799506:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664651008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774907.799509:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799524:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664650240:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774907.799552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774907.799594:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799609:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2f100 x1862032664650368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.799623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664650368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.799633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799636:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774907.799647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774907.799668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2df80 x1862032664650496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.799694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664650496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.799710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774907.799724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650624:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774907.799746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2ca80 x1862032664650624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.799770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664650624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.799778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774907.799792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774907.799813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799824:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2d500 x1862032664650752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.799836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664650752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.799845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799848:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799852:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664650752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774907.799859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664650880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774907.799894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2f480 x1862032664650880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.799919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664650880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.799929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.799948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.799953:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664650880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774907.799961:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774907.799983:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774907.799996:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774907/real 1775774907] req@ffff982974b2e300 x1862032664651008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774907.800012:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664651008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774907.800023:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774907.800027:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774907.800031:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774908.311233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774908.567289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774909.335142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774909.591245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774910.359162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774910.615380:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774911.383160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774911.639185:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774912.407146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774912.663242:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774912.919135:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919143:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774912.919148:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919157:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919167:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919172:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774912.919207:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664651136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774912.919230:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919233:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774912.919234:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919237:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919239:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919242:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919249:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774912.919257:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664651264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774912.919264:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919266:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774912.919268:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919271:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919273:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919276:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919280:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774912.919287:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664651392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774912.919294:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664651136:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775774912.919296:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919303:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774912.919304:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919307:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919309:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919312:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919316:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774912.919323:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664651520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774912.919329:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919332:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774912.919333:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919336:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919338:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919340:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774912.919351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664651648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774912.919359:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774912.919360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774912.919366:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774912.919369:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919372:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919376:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774912.919385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664651776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774912.919392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774912.919396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774912.919398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774912.919403:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe1180 x1862032664651136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774912.919406:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16680 00000100:00080000:1.0:1775774912.919410:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774912.919418:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774912.919426:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774912.919427:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664651136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919438:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775774912.919443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664651904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774912.919445:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919451:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664651136:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774912.919462:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774912.919482:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919493:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe1500 x1862032664651264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919501:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664651264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919506:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919509:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919511:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774912.919530:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774912.919540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe1c00 x1862032664651392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919554:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664651392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919564:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919566:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774912.919572:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774912.919583:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe0e00 x1862032664651520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919595:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664651520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919600:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919602:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919604:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774912.919607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774912.919617:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe0a80 x1862032664651648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664651648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919643:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664651648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774912.919653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774912.919672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe1f80 x1862032664651776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664651776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919704:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774912.919716:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774912.919730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774912.919738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774912/real 1775774912] req@ffff982981fe3b80 x1862032664651904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774912.919745:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664651904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774912.919750:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774912.919752:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774912.919754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664651904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774913.431183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774913.687242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774914.455275:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774914.711206:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774915.479234:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774915.735306:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774916.503197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774916.759300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774917.527208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774917.783215:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774918.039144:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039151:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774918.039155:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039163:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039174:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039180:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774918.039216:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664652032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774918.039239:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039242:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774918.039243:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039246:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039248:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039250:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774918.039258:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774918.039266:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664652160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774918.039272:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039275:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774918.039277:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039280:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039282:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039285:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774918.039290:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774918.039300:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664652032:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774918.039305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664652288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774918.039317:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039321:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774918.039323:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039328:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039333:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039338:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774918.039344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774918.039357:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774918.039358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664652416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774918.039367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774918.039372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039375:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039378:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039381:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774918.039386:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774918.039390:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe0380 x1862032664652032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774918.039395:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664652544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774918.039410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775774918.039415:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664652032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774918.039416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039423:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039429:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774918.039433:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774918.039435:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775774918.039437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774918.039440:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775774918.039447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664652672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774918.039448:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664652032:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774918.039459:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652160:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775774918.039464:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774918.039468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774918.039470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774918.039476:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16685 00000100:00080000:1.0:1775774918.039481:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775774918.039481:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774918.039489:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774918.039494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774918.039502:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe0000 x1862032664652160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774918.039510:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664652800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774918.039516:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664652160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039529:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039532:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039534:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774918.039556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774918.039570:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774918.039576:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe1880 x1862032664652288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774918.039584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664652288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039598:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774918.039602:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774918.039613:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774918.039619:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe2300 x1862032664652416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774918.039626:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664652416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039646:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774918.039651:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774918.039665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774918.039672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe2680 x1862032664652544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774918.039679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664652544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039691:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664652544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774918.039697:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664652672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774918.039714:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774918.039725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe2a00 x1862032664652672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774918.039738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664652672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664652672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774918.039761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664652800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774918.039776:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774918.039784:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774918/real 1775774918] req@ffff982981fe3480 x1862032664652800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774918.039792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664652800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774918.039798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774918.039800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774918.039802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664652800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774918.551177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774918.807245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774919.575168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774919.831228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774920.599151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774920.855189:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774921.623205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774921.879264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774922.647191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774922.903246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774923.159204:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159214:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774923.159218:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159238:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774923.159284:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664652928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774923.159305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774923.159309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159317:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774923.159325:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774923.159333:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664653056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774923.159339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159342:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774923.159343:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159353:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664652928:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775774923.159354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774923.159363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664653184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774923.159369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774923.159373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159376:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159378:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159380:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774923.159383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774923.159392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664653312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774923.159397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774923.159400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774923.159410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774923.159416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664653440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774923.159422:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774923.159427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774923.159432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774923.159437:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159441:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159445:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774923.159451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774923.159457:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2d180 x1862032664652928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774923.159463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664653568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774923.159485:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774923.159488:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774923.159490:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774923.159490:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664652928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774923.159497:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16690 00000100:00080000:0.0:1775774923.159503:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774923.159506:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774923.159507:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775774923.159514:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774923.159514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159518:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664652928:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774923.159527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664653696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774923.159527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774923.159551:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159561:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2c380 x1862032664653056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159569:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664653056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774923.159597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774923.159607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159613:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2e680 x1862032664653184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664653184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774923.159636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653312:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774923.159654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2e300 x1862032664653312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664653312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159687:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774923.159691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653440:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774923.159704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159711:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2f480 x1862032664653440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664653440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774923.159728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664653568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774923.159737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159742:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2d500 x1862032664653568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664653568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159755:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664653568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774923.159758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664653696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774923.159788:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774923.159796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774923/real 1775774923] req@ffff982974b2ca80 x1862032664653696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774923.159803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664653696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774923.159807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774923.159809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774923.159811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664653696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774923.671163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774923.927184:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774924.695138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774924.951232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774925.719182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774925.975264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774926.743148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774926.999192:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774927.767169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774928.023257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774928.279235:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774928.279248:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774928.279253:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774928.279266:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279280:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.279287:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774928.279340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664653824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774928.279383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774928.279388:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774928.279390:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774928.279468:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664653824:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774928.279502:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.279523:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774928.279538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775774928.279551:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774928.279575:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664653952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774928.279597:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775774928.279600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2df80 x1862032664653824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774928.279608:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774928.279612:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774928.279627:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279634:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.279639:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774928.279644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664653824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774928.279648:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774928.279669:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774928.279686:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664654080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774928.279704:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774928.279706:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.279715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664653824:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774928.279716:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774928.279721:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774928.279728:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279734:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.279738:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774928.279745:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774928.279754:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774928.279764:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664654208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774928.279778:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774928.279782:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774928.279784:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775774928.279788:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774928.279795:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279800:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.279808:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774928.279810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2f100 x1862032664653952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774928.279816:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774928.279840:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664654336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774928.279844:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664653952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774928.279886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774928.279889:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774928.279894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775774928.279900:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774928.279905:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775774928.279905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664653952:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774928.279918:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.279923:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774928.279925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654080:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774928.279933:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774928.279941:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774928.279955:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664654464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775774928.279961:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774928.279977:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774928.279982:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775774928.279983:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2fb80 x1862032664654080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774928.279985:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774928.280000:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16695 00000100:00080000:0.0:1775774928.280009:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774928.280017:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774928.280017:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664654080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774928.280027:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775774928.280048:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774928.280051:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664654592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774928.280056:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.280119:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774928.280146:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774928.280174:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774928.280192:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2df80 x1862032664654208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774928.280205:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664654208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774928.280214:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774928.280217:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.280220:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774928.280226:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774928.280245:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774928.280255:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2ca80 x1862032664654336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774928.280266:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664654336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774928.280273:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774928.280277:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.280280:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774928.280287:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664654464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774928.280304:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774928.280313:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2d500 x1862032664654464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774928.280324:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664654464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774928.280331:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774928.280335:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.280338:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664654464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774928.280344:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664654592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774928.280361:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774928.280371:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774928/real 1775774928] req@ffff982974b2f480 x1862032664654592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774928.280382:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664654592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774928.280390:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774928.280394:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774928.280396:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664654592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774928.791143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774929.047232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774929.815186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775774930.071201:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774930.839164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774931.095211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774931.863175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774932.119249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774932.887163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774933.143191:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774933.399189:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399199:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774933.399204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399216:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399228:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399236:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774933.399274:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664654720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774933.399302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399305:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774933.399307:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399311:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399314:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399318:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774933.399329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774933.399342:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664654848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774933.399349:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664654720:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774933.399357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399361:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774933.399362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399367:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399371:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399375:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774933.399380:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774933.399392:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664654976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774933.399401:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399404:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774933.399406:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399409:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16700 00000800:00000100:3.0:1775774933.399414:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774933.399415:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399421:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774933.399426:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774933.399436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664655104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774933.399445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399449:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775774933.399450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b5500 x1862032664654720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774933.399451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399460:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399465:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399468:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399471:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664654720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775774933.399474:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775774933.399483:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399487:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775774933.399488:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664655232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774933.399491:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664654720:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774933.399502:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774933.399504:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774933.399506:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399509:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399512:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774933.399515:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399518:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654848:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775774933.399519:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774933.399530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664655360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774933.399537:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775774933.399541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775774933.399542:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774933.399544:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774933.399548:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16700 00000100:00080000:2.0:1775774933.399551:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775774933.399554:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b4000 x1862032664654848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774933.399555:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774933.399560:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774933.399570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664654848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774933.399573:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664655488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774933.399606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399611:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399614:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774933.399619:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774933.399633:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b6680 x1862032664654976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774933.399646:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664654976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774933.399655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664654976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774933.399662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774933.399672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b4e00 x1862032664655104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774933.399683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664655104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774933.399687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774933.399693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774933.399702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b7b80 x1862032664655232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774933.399712:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664655232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774933.399716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774933.399730:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664655360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774933.399738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b7100 x1862032664655360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774933.399749:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664655360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774933.399753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664655360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774933.399759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664655488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774933.399767:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774933.399772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774933/real 1775774933] req@ffff9829764b6a00 x1862032664655488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774933.399778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664655488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774933.399782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774933.399783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774933.399785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664655488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774933.911179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774934.167244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774934.935116:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774935.192173:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774935.959188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774936.215283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774936.983204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774937.239301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774938.007169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774938.263230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774938.519217:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774938.519230:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774938.519235:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.519247:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.519259:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774938.519266:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774938.519315:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664655616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774938.519348:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774938.519351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774938.519352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.519356:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.519358:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774938.519362:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774938.519372:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774938.519387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664655744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774938.519401:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774938.519405:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774938.519407:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.519412:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.519417:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774938.519424:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664655616:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774938.519425:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774938.519435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774938.519446:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664655872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774938.519511:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774938.519552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9df80 x1862032664655616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774938.519570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664655616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774938.519577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.519580:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.519584:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664655616:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774938.519622:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774938.519645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774938.519656:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9e680 x1862032664655744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774938.519663:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664655744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774938.519669:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.519671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.519673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655744:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774938.520649:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774938.520652:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774938.520654:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.520658:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.520663:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774938.520668:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774938.520676:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774938.520681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774938.520698:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664656000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774938.520705:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774938.520706:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774938.520709:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774938.520711:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.520714:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.520717:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774938.520718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9ea00 x1862032664655872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774938.520722:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774938.520728:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774938.520733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664655872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774938.520738:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664656128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774938.520746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774938.520752:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774938.520754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774938.520756:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775774938.520760:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664655872:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774938.520760:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.520767:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.520770:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774938.520773:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774938.520776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656000:192.168.201.118@tcp:8:kworker.0 00000008:00100000:3.0:1775774938.520776:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774938.520786:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664656256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774938.520793:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775774938.520796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774938.520798:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774938.520801:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774938.520805:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16706 00000100:00080000:3.0:1775774938.520808:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774938.520810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9c000 x1862032664656000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774938.520813:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774938.520819:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774938.520824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664656000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774938.520828:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664656384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774938.520841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.520847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.520851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774938.520858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656128:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774938.520878:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774938.520889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9d180 x1862032664656128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774938.520896:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664656128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774938.520901:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.520903:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.520905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774938.520913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664656256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774938.520925:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774938.520931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9f800 x1862032664656256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774938.520937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664656256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774938.520941:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.520943:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.520945:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664656256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774938.520948:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664656384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774938.520956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774938.520961:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774938/real 1775774938] req@ffff982980f9c700 x1862032664656384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774938.520967:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664656384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774938.520971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774938.520973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774938.520974:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664656384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774939.031194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774939.287286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774940.055237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774940.311250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774941.079175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774941.335256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774942.103167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774942.359225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774943.127207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774943.383293:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774943.639188:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639195:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774943.639199:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639206:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639215:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774943.639220:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774943.639250:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664656512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639272:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639275:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774943.639277:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639281:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639284:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774943.639288:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639296:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774943.639308:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664656640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639319:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639322:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774943.639323:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639326:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774943.639331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664656512:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775774943.639334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774943.639350:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664656768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639358:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774943.639359:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639361:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639363:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774943.639366:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774943.639376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664656896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639383:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774943.639384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639386:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639390:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774943.639392:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774943.639402:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774943.639402:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664657024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639436:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774943.639437:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639440:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639443:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774943.639447:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774943.639455:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b6a00 x1862032664656512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775774943.639460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664657152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774943.639477:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774943.639480:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774943.639481:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774943.639483:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16711 00000100:00080000:2.0:1775774943.639485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774943.639488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664656512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774943.639489:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774943.639494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774943.639502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664657280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774943.639506:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639521:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664656512:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774943.639529:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774943.639551:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639562:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b7100 x1862032664656640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664656640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639582:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639585:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774943.639610:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774943.639626:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b7b80 x1862032664656768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664656768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639651:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639657:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774943.639663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774943.639678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b4e00 x1862032664656896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664656896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639712:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664656896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774943.639718:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774943.639737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b6680 x1862032664657024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664657024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774943.639774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664657152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774943.639791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b4000 x1862032664657152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664657152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639824:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664657152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774943.639828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664657280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774943.639844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774943.639853:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774943/real 1775774943] req@ffff9829764b5500 x1862032664657280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774943.639863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664657280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774943.639870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774943.639873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774943.639875:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664657280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774944.151211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774944.407299:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774945.175201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774945.431298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774946.199167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774946.455269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774947.223195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774947.479268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774948.247142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774948.503283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774948.759297:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759308:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774948.759313:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759334:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759347:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759353:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774948.759388:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664657408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774948.759419:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759422:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774948.759424:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759427:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759430:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759433:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774948.759450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664657536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774948.759458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774948.759463:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759465:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759467:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759470:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774948.759482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664657664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774948.759485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664657408:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774948.759491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759496:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774948.759497:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759500:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759503:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759508:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774948.759516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664657792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774948.759523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759525:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774948.759527:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759529:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759531:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759533:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759536:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774948.759542:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664657920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774948.759548:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759551:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774948.759552:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759554:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759556:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759560:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774948.759562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759564:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774948.759572:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664658048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774948.759579:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774948.759582:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774948.759584:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774948.759587:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16716 00000100:00080000:3.0:1775774948.759590:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774948.759593:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774948.759596:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774948.759605:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664658176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774948.759606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9d880 x1862032664657408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664657408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759645:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664657408:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774948.759661:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774948.759682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759692:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9e300 x1862032664657536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664657536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774948.759742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774948.759762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9ce00 x1862032664657664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664657664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774948.759809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657792:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774948.759828:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9c700 x1862032664657792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664657792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759849:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759851:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774948.759856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774948.759866:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9f800 x1862032664657920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759879:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664657920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759883:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759887:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664657920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774948.759890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774948.759900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9d180 x1862032664658048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664658048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774948.759923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774948.759932:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774948.759938:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774948/real 1775774948] req@ffff982980f9c000 x1862032664658176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774948.759945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664658176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774948.759950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774948.759951:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774948.759953:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774949.271179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774949.527255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774950.295213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774950.551231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774951.319179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774951.575265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774952.343173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774952.599218:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774953.367184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774953.623278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774953.879186:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879196:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774953.879201:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774953.879212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879225:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879244:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774953.879285:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664658304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774953.879310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879314:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774953.879316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774953.879321:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879326:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879332:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774953.879343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774953.879349:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664658304:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774953.879360:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664658432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774953.879380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774953.879386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774953.879391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879396:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879401:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774953.879407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775774953.879412:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774953.879424:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664658560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774953.879435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775774953.879443:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9ea00 x1862032664658304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774953.879443:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774953.879452:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879460:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879465:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879468:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664658304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774953.879474:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775774953.879478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879483:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879489:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664658304:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774953.879490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664658688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774953.879503:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879507:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774953.879509:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774953.879513:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658432:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774953.879515:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879522:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879527:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774953.879533:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774953.879536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879549:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9e680 x1862032664658432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774953.879553:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664658816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774953.879569:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774953.879574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664658432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774953.879575:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774953.879579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774953.879590:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774953.879590:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:0.0:1775774953.879597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775774953.879597:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774953.879603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658432:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774953.879606:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658560:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775774953.879613:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774953.879629:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664658944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774953.879634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879644:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9df80 x1862032664658560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774953.879646:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774953.879656:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774953.879659:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774953.879664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664658560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774953.879666:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16721 00000100:00080000:3.0:1775774953.879673:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774953.879678:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774953.879682:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774953.879696:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664659072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774953.879703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879719:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774953.879731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774953.879751:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9f100 x1862032664658688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774953.879768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664658688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774953.879773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774953.879788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774953.879799:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9c380 x1862032664658816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774953.879813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664658816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774953.879817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664658816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774953.879826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774953.879836:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9fb80 x1862032664658944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774953.879856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664658944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774953.879864:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879867:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664658944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774953.879877:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774953.879898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774953.879909:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774953/real 1775774953] req@ffff982980f9d500 x1862032664659072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774953.879917:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664659072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774953.879923:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774953.879925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774953.879928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774954.391176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774954.647253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774955.415198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774955.671256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774956.439179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774956.695287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774957.463219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774957.719292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774958.487228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774958.743295:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774958.999169:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999180:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774958.999186:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999197:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999210:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774958.999265:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664659200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999288:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999291:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774958.999293:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999296:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999299:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774958.999310:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774958.999319:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664659328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999331:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774958.999333:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999336:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999339:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999343:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774958.999348:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774958.999362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664659200:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774958.999363:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664659456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999386:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774958.999393:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999397:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999402:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999406:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774958.999412:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774958.999426:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664659584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999437:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999442:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774958.999444:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999449:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999456:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999460:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774958.999465:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775774958.999474:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775774958.999479:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664659712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999488:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999491:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774958.999493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999498:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999500:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774958.999504:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774958.999509:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774958.999519:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe3480 x1862032664659200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774958.999520:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664659840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774958.999556:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774958.999563:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775774958.999567:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664659200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774958.999568:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774958.999582:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16726 00000100:00080000:1.0:1775774958.999595:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774958.999600:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775774958.999601:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775774958.999613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:1.0:1775774958.999615:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774958.999620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664659200:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774958.999630:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774958.999633:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664659968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774958.999665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999679:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe2a00 x1862032664659328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664659328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774958.999696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774958.999699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774958.999701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774958.999725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774958.999745:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe2680 x1862032664659456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664659456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774958.999779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774958.999782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774958.999786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774958.999794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774958.999815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999824:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe2300 x1862032664659584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664659584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774958.999838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774958.999840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774958.999842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774958.999846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774958.999860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999865:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe1880 x1862032664659712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664659712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774958.999883:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774958.999885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774958.999886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664659712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774958.999890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774958.999900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999906:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe0000 x1862032664659840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664659840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774958.999927:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774958.999931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774958.999934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774958.999941:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774958.999963:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774958.999974:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774958/real 1775774958] req@ffff982981fe0380 x1862032664659968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774958.999985:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664659968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774959.000174:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774959.000181:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774959.000187:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664659968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774959.511187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774959.767275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774960.535187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774960.791287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774961.559167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774961.815268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774962.583185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774962.839252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774963.607163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774963.863244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774964.119178:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119188:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774964.119193:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119203:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119225:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119231:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774964.119273:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664660096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774964.119301:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119304:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774964.119305:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119309:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119312:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119315:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119325:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774964.119340:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664660224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774964.119342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664660096:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774964.119352:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774964.119356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119358:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119361:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119363:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119367:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774964.119376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664660352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774964.119383:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774964.119387:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119390:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119392:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119395:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774964.119406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664660480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774964.119414:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774964.119420:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119425:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119429:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119434:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774964.119453:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664660608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774964.119464:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119469:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774964.119473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774964.119474:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660224:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774964.119481:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774964.119489:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774964.119506:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664660736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774964.119510:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660352:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775774964.119527:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774964.119541:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774964.119543:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774964.119550:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16731 00000100:00080000:3.0:1775774964.119555:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775774964.119560:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660480:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775774964.119562:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774964.119570:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774964.119584:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664660864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774964.119606:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774964.119629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664660736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774964.119647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664660864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774964.631225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774964.887263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774965.655132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774965.911243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774966.679190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774966.935243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775774967.191420:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:0.0:1775774967.191471:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775774967.191492:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191551:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191562:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191571:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191579:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191587:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775774967.191597:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775774967.191628:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9c380 x1862032664660736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664660736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191676:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191681:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664660736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774967.191706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9f100 x1862032664660864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664660864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191735:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664660864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774967.191742:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9f480 x1862032664660224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664660224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774967.191776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9dc00 x1862032664660352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664660352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191797:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774967.191812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9d500 x1862032664660480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664660480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191832:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191835:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774967.191846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9fb80 x1862032664660608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191857:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664660608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191865:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191872:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664660608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774967.191879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774964/real 1775774967] req@ffff982980f9ed80 x1862032664660096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774967.191890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664660096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774967.191897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774967.191900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774967.191903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664660096:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774967.703129:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774967.959256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774968.727203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774968.983243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774969.239173:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774969.239180:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774969.239184:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239192:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239203:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239208:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774969.239239:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664660992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774969.239263:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774969.239267:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774969.239269:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239273:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239277:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239282:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774969.239291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774969.239304:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664661120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774969.239313:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774969.239317:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664660992:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774969.239317:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774969.239319:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239326:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239330:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774969.239335:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774969.239343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664661248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774969.239352:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774969.239355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774969.239356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239360:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239363:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239367:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774969.239372:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774969.239381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664661376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774969.239388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774969.239392:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774969.239393:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774969.239396:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774969.239397:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239400:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774969.239407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774969.239415:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664661504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774969.239424:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775774969.239425:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9ed80 x1862032664660992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774969.239427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774969.239430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239433:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239436:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239439:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239442:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664660992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774969.239444:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774969.239453:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775774969.239453:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664661632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774969.239457:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239461:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664660992:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774969.239462:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774969.239466:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774969.239467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774969.239471:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16736 00000100:00080000:3.0:1775774969.239473:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774969.239477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774969.239481:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774969.239485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774969.239489:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664661760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774969.239507:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239519:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9fb80 x1862032664661120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239533:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664661120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239546:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239550:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774969.239556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774969.239577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9d500 x1862032664661248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239601:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664661248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239610:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774969.239622:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774969.239645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239657:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9dc00 x1862032664661376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664661376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239678:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239681:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774969.239690:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774969.239726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9f480 x1862032664661504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664661504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239761:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664661504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774969.239766:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664661632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774969.239786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9f100 x1862032664661632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664661632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664661632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774969.239841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664661760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774969.239862:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774969.239872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774969/real 1775774969] req@ffff982980f9c380 x1862032664661760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774969.239889:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664661760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774969.239896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774969.239898:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774969.239901:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664661760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774969.751213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774970.007221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774970.775186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774971.031261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774971.799161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774972.055204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774972.823165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774973.079229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774973.847173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774974.103186:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774974.359185:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359194:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774974.359198:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359207:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359218:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359224:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774974.359257:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664661888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774974.359288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359291:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359294:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359296:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359303:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774974.359312:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664662016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359319:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774974.359323:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359326:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359328:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774974.359336:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664661888:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774974.359343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664662144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359355:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359358:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774974.359359:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359362:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359364:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359366:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359370:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774974.359378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664662272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359384:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774974.359388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359390:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359396:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775774974.359401:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774974.359413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664662400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359419:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774974.359422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359426:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359429:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359432:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774974.359440:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe3b80 x1862032664661888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774974.359445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664662528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774974.359467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774974.359469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774974.359470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774974.359476:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16741 00000100:00080000:1.0:1775774974.359480:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774974.359483:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774974.359486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774974.359491:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664661888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775774974.359495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664662656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774974.359517:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359522:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359529:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664661888:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774974.359538:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774974.359559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe1f80 x1862032664662016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359582:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664662016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359591:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359598:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774974.359623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774974.359647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe0a80 x1862032664662144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664662144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359673:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774974.359685:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662272:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774974.359707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe0e00 x1862032664662272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359731:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664662272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774974.359750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774974.359764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe1c00 x1862032664662400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664662400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774974.359791:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664662528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774974.359800:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe1500 x1862032664662528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664662528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664662528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774974.359824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664662656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774974.359833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774974.359838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774974/real 1775774974] req@ffff982981fe1180 x1862032664662656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774974.359854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664662656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775774999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774974.359859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774974.359861:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774974.359863:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664662656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775774974.871153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774975.127238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774975.895167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774976.151257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774976.919153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774977.175264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774977.943118:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774978.199288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774978.967208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774979.223225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775774979.479173:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479182:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775774979.479186:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479195:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479205:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775774979.479247:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664662784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774979.479271:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774979.479275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479283:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479291:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774979.479300:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664662912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774979.479308:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774979.479312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479315:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479317:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479323:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774979.479328:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664662784:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775774979.479333:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664663040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774979.479340:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479343:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775774979.479344:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479347:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479349:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479352:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774979.479364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664663168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774979.479371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775774979.479375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479377:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479379:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479382:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775774979.479392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664663296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774979.479402:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775774979.479406:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479410:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775774979.479411:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775774979.479414:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479417:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479420:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774979.479431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664663424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775774979.479438:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775774979.479441:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775774979.479442:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775774979.479445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe1180 x1862032664662784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775774979.479448:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16746 00000100:00080000:1.0:1775774979.479453:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775774979.479458:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775774979.479465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775774979.479473:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664663552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774979.479476:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664662784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479497:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479501:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479505:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664662784:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774979.479511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774979.479532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479540:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe1500 x1862032664662912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479548:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664662912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479552:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479554:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664662912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774979.479574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774979.479585:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479592:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe1c00 x1862032664663040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664663040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774979.479625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774979.479647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe0e00 x1862032664663168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664663168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774979.479681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774979.479696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479702:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe0a80 x1862032664663296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664663296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774979.479721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664663424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774979.479731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe1f80 x1862032664663424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664663424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479748:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664663424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774979.479756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664663552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774979.479773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774979.479813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774979/real 1775774979] req@ffff982981fe3b80 x1862032664663552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774979.479829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664663552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774979.479837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774979.479839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774979.479841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664663552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774979.991177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774980.247241:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774981.015154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774981.271314:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774982.039162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774982.295253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774983.063129:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774983.319221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775774984.087167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774984.343275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774984.599252:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599262:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774984.599267:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599278:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599292:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599304:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774984.599350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664663680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774984.599386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599390:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599392:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774984.599413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664663808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599423:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774984.599424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599427:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599429:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599432:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774984.599441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664663936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599448:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599451:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775774984.599452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664663680:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775774984.599452:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599457:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599466:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774984.599477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664664064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599483:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774984.599487:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599489:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599492:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599494:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774984.599505:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664664192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599511:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774984.599514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774984.599515:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599518:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599520:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774984.599528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664664320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774984.599538:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774984.599541:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774984.599542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775774984.599548:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774984.599548:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16752 00000100:00080000:3.0:1775774984.599553:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774984.599556:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774984.599559:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774984.599565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664664448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774984.599599:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9df80 x1862032664663680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.599623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664663680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.599636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.599640:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.599645:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664663680:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774984.599654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774984.599680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.599711:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9e680 x1862032664663808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.599726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664663808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.599735:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.599739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.599742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774984.599773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774984.599797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.599808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9ea00 x1862032664663936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.599822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664663936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.599831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.599835:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.599839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664663936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774984.599845:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774984.599867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.599879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9c000 x1862032664664064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.599892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664664064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.599901:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.599904:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.599908:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774984.599914:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774984.599934:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.599953:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9d180 x1862032664664192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.599966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664664192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.599974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.599978:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.599981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774984.599987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664664320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774984.600005:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.600016:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9f800 x1862032664664320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.600027:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664664320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.600036:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.600039:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.600042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664664320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774984.600047:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664664448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774984.600118:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774984.600129:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774984/real 1775774984] req@ffff982980f9c700 x1862032664664448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774984.600141:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664664448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774984.600149:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774984.600153:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774984.600156:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664664448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775774985.111129:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774985.367247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774986.135230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774986.391307:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774987.159152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774987.415218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775774988.183204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774988.439231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774989.207192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774989.463195:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775774989.719181:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719191:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775774989.719198:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719208:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00080000:0.0:1775774989.719221:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719228:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774989.719266:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664664576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774989.719300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774989.719307:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719311:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00080000:0.0:1775774989.719316:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719321:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774989.719330:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774989.719345:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664664704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774989.719345:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664664576:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775774989.719359:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719362:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774989.719363:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719366:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00080000:0.0:1775774989.719369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719371:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774989.719375:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774989.719383:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664664832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774989.719395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775774989.719399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719401:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00080000:0.0:1775774989.719404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719406:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774989.719409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774989.719419:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774989.719420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664664960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775774989.719428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775774989.719433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719436:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00080000:0.0:1775774989.719438:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775774989.719444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775774989.719451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664665088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774989.719458:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2e300 x1862032664664576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774989.719461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719472:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775774989.719477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719484:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16757 00000100:00100000:3.0:1775774989.719486:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664664576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774989.719492:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774989.719498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774989.719502:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775774989.719505:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719510:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664664576:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775774989.719511:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774989.719528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664665216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774989.719534:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664704:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775774989.719547:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775774989.719551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775774989.719554:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775774989.719559:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16757 00000800:00000100:3.0:1775774989.719560:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775774989.719566:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775774989.719573:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719575:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2e680 x1862032664664704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775774989.719582:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774989.719587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664664704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775774989.719599:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664665344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774989.719600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774989.719624:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774989.719642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2c380 x1862032664664832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774989.719661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664664832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774989.719671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774989.719677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664960:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774989.719689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2d180 x1862032664664960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774989.719702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664664960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774989.719710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719712:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664664960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774989.719715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774989.719724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2f480 x1862032664665088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774989.719736:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664665088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774989.719744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774989.719757:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664665216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774989.719767:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2d500 x1862032664665216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774989.719780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664665216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774989.719788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664665216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774989.719794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664665344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774989.719812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774989.719819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774989/real 1775774989] req@ffff982974b2ca80 x1862032664665344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774989.719826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664665344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774989.719831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774989.719833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774989.719835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664665344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774990.231180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774990.487240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774991.255177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774991.511210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774992.279159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774992.535268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774993.303204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774993.559247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774994.327156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774994.583249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775774994.839197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839205:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775774994.839209:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839217:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839234:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775774994.839275:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664665472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774994.839298:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839301:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774994.839302:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839306:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839308:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839310:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839318:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774994.839327:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664665600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774994.839335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774994.839339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839343:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839346:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839349:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774994.839354:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664665472:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774994.839357:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664665728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774994.839368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775774994.839372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839374:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839376:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839378:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839381:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775774994.839390:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664665856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774994.839396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775774994.839400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839414:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839417:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839419:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775774994.839426:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775774994.839429:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664665984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775774994.839436:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775774994.839439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775774994.839440:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839443:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00100000:3.0:1775774994.839458:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b6d80 x1862032664665472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839477:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664665472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775774994.839482:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839486:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775774994.839499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664666112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774994.839501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775774994.839511:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774994.839513:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775774994.839532:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775774994.839534:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775774994.839537:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16762 00000100:00080000:2.0:1775774994.839540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775774994.839544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775774994.839549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774994.839552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664665472:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775774994.839560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664666240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775774994.839565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774994.839600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839613:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b4a80 x1862032664665600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839626:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664665600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839640:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839643:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774994.839670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774994.839689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839700:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b5180 x1862032664665728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664665728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774994.839746:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774994.839766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b5c00 x1862032664665856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839789:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664665856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839808:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774994.839818:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775774994.839837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b4700 x1862032664665984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839860:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664665984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839872:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839875:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664665984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775774994.839881:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664666112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774994.839898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839907:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b5f80 x1862032664666112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664666112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664666112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774994.839940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664666240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775774994.839959:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775774994.839969:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774994/real 1775774994] req@ffff9829764b5500 x1862032664666240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775774994.839981:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664666240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774994.839987:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775774994.839989:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775774994.839991:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664666240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774995.351167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774995.607251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774996.375185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774996.631284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774997.399158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774997.655301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774998.423139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774998.679251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775774999.447162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775774999.703224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775774999.959199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775774999.959212:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959221:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959232:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959239:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775774999.959271:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664666368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774999.959297:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959300:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774999.959301:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959305:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959307:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959310:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774999.959318:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774999.959326:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664666496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774999.959333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774999.959337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959340:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959342:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774999.959347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775774999.959351:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664666368:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775774999.959356:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664666624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774999.959367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775774999.959371:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959374:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959376:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959378:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774999.959382:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774999.959391:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664666752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775774999.959397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775774999.959401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959406:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959408:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774999.959411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775774999.959417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664666880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774999.959423:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775774999.959442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775774999.959447:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959451:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959455:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959458:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775774999.959471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775774999.959479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664667008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775774999.959486:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9ce00 x1862032664666368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775774999.959489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775774999.959497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775774999.959500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775774999.959507:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16767 00000100:00080000:3.0:1775774999.959510:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775774999.959513:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959516:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664666368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775774999.959518:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775774999.959529:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959537:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664666368:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775774999.959544:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775774999.959545:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664667136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775774999.959565:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959583:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9e300 x1862032664666496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959600:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664666496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774999.959628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774999.959641:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959647:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9d880 x1862032664666624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664666624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959666:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774999.959671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774999.959682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9c700 x1862032664666752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664666752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774999.959707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775774999.959716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9f800 x1862032664666880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664666880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664666880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775774999.959745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774999.959754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9d180 x1862032664667008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664667008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775774999.959778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775774999.959794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775774999.959800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775774999/real 1775774999] req@ffff982980f9c000 x1862032664667136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775774999.959807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664667136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775774999.959813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775774999.959815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775774999.959817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775000.471171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775000.727243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775001.495196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775001.751235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775002.519159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775002.775235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775003.543236:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775003.799311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775004.567166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775004.823253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775005.079139:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775005.079146:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775005.079150:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079158:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079171:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079178:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775005.079215:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664667264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775005.079240:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775005.079244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775005.079246:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079250:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079254:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079257:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079266:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775005.079276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664667392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775005.079283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775005.079286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775005.079288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079290:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079293:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079295:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079298:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775005.079305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664667520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775005.079313:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775005.079317:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664667264:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775005.079317:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775005.079318:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079325:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079327:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079330:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079334:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775005.079362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664667648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775005.079383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775005.079387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775005.079389:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079394:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079399:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079404:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775005.079423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664667776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775005.079430:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775005.079442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775005.079446:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775005.079448:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775005.079454:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00080000:3.0:1775775005.079460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775005.079466:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079472:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775005.079473:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9ea00 x1862032664667264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775005.079489:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664667904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775005.079493:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664667264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775005.079509:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775005.079513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775005.079513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775005.079516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775005.079519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775005.079525:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16772 00000100:00100000:0.0:1775775005.079526:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664667264:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775005.079530:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775005.079535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667392:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775005.079543:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775005.079550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775005.079560:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775005.079568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664668032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775005.079573:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9e680 x1862032664667392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664667392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079603:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079606:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775005.079631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775005.079642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775005.079648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9df80 x1862032664667520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664667520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079661:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775005.079668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775005.079679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775005.079685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9c380 x1862032664667648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664667648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775005.079704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775005.079714:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775005.079720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9f100 x1862032664667776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664667776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664667776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775005.079739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775005.079748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775005.079753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9f480 x1862032664667904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664667904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664667904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775005.079772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775005.079782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775005.079787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775005/real 1775775005] req@ffff982980f9dc00 x1862032664668032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775005.079794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664668032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775005.079800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775005.079802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775005.079804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775005.591212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775005.847313:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775006.615174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775006.871246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775007.639253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775007.895281:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775008.663221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775008.919293:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775009.687209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775009.943320:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775010.199243:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199267:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775010.199272:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199285:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199316:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199323:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775010.199370:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664668160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775010.199404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775010.199409:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199412:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199414:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775010.199427:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775010.199442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664668288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775010.199450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775010.199455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199458:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199468:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664668160:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775010.199470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775010.199483:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664668416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775010.199490:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775010.199494:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199497:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199502:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199506:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775010.199512:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775010.199526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664668544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775010.199537:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199541:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775010.199543:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775010.199549:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775010.199552:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199558:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199563:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775010.199569:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775010.199583:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664668672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775010.199590:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775010.199591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b4000 x1862032664668160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775010.199597:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775010.199600:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199608:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199633:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775010.199637:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664668160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775010.199643:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775010.199660:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664668800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775010.199670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775010.199696:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775010.199701:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775010.199702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664668160:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775010.199703:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775010.199711:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16777 00000100:00080000:2.0:1775775010.199714:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775010.199715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668288:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775010.199720:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775010.199723:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775010.199735:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664668928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775010.199743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b6680 x1862032664668288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664668288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199776:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775010.199809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775010.199822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199829:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b4e00 x1862032664668416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664668416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199852:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775010.199856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775010.199868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b7b80 x1862032664668544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199882:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664668544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199891:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775010.199895:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775010.199905:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b7100 x1862032664668672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199918:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664668672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199923:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199926:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664668672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775010.199929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775010.199939:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b6a00 x1862032664668800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199952:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664668800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199957:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199959:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199961:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775010.199964:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775010.199973:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775010.199979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775010/real 1775775010] req@ffff9829764b5500 x1862032664668928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775010.199986:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664668928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775010.199992:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775010.199993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775010.199995:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664668928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775010.711252:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775010.967263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775011.735201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775011.991256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775012.759251:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775013.015228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775013.783201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775014.039257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775014.807184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775015.063339:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775015.319197:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775015.319221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319242:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319247:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775015.319282:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664669056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775015.319306:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775015.319352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319358:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319364:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775015.319401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664669184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775015.319425:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664669056:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775015.319431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319443:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775015.319445:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319449:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319452:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319455:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775015.319478:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664669312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775015.319485:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319488:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775015.319490:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319493:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319496:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319500:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775015.319515:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664669440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775015.319521:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319523:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775015.319525:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319528:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319532:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319535:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319538:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775015.319546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664669568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775015.319552:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775015.319564:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319567:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775015.319569:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319577:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775015.319580:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319585:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775015.319594:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664669696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775015.319603:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775015.319606:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775015.319608:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775015.319612:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16782 00000100:00080000:1.0:1775775015.319618:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775015.319622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe0380 x1862032664669056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775015.319627:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775015.319639:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775015.319667:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664669824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775015.319689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664669056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.319704:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.319710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.319715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664669056:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775015.319767:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775015.319812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.319869:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe0000 x1862032664669184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.319893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664669184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.319907:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.319913:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.319919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775015.319930:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775015.319999:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.320012:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe1880 x1862032664669312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.320023:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664669312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.320031:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.320034:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.320037:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775015.320043:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669440:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775015.320061:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.320086:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe2300 x1862032664669440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.320099:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664669440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.320191:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.320217:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.320220:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775015.320227:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775015.320248:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.320268:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe2680 x1862032664669568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.320290:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664669568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.320302:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.320306:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.320310:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664669568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775015.320319:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664669696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775015.320346:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.320357:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe2a00 x1862032664669696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.320367:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664669696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.320374:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.320377:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.320381:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664669696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775015.320404:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664669824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775015.320425:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775015.320435:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775015/real 1775775015] req@ffff982981fe3480 x1862032664669824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775015.320447:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664669824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775015.320456:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775015.320459:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775015.320462:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664669824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775015.831265:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775016.087300:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775016.855208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775017.111389:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775017.879159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775018.135225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775018.903162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775019.159258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775019.927174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775020.183270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775020.439192:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775020.439200:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775020.439205:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439215:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439225:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439230:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775020.439263:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664669952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439287:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775020.439290:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775020.439291:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439294:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439300:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775020.439308:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775020.439318:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664670080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775020.439328:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775020.439329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439339:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664669952:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775020.439343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775020.439353:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664670208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775020.439374:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775020.439376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439379:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439382:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439385:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775020.439388:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775020.439397:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664670336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439403:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775020.439408:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775020.439410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775020.439412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439416:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439419:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439422:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775020.439426:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775020.439433:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664670464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439440:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775020.439442:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775020.439445:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775020.439447:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9d500 x1862032664669952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775020.439450:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439455:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439460:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775020.439467:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775020.439477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664670592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775020.439483:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775020.439486:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664669952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775020.439487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775020.439490:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775020.439498:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16787 00000100:00080000:3.0:1775775020.439502:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775020.439506:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775020.439508:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775020.439510:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775020.439514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439521:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664669952:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775020.439521:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664670720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775020.439535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775020.439557:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439567:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9fb80 x1862032664670080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439576:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664670080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439581:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439583:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439586:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775020.439605:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775020.439616:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9ed80 x1862032664670208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664670208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775020.439642:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775020.439654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439660:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9dc00 x1862032664670336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664670336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439683:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439685:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775020.439689:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775020.439700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9f480 x1862032664670464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664670464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775020.439725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664670592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775020.439735:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9f100 x1862032664670592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664670592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664670592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775020.439760:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664670720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775020.439769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775020.439774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775020/real 1775775020] req@ffff982980f9c380 x1862032664670720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775020.439780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664670720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775020.439785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775020.439786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775020.439788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664670720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775020.951190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775021.207253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775021.975192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775022.231244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775022.999181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775023.255253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775024.023154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775024.279232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775025.047170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775025.303352:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775025.559118:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559128:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775025.559133:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559144:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559158:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559165:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775025.559195:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664670848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775025.559226:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559232:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775025.559234:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559239:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559243:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559248:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775025.559257:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775025.559269:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664670976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775025.559277:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664670848:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775025.559280:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559285:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775025.559287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559291:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559294:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559297:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775025.559301:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775025.559310:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664671104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775025.559317:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559320:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775025.559321:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559323:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559328:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559330:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775025.559334:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775025.559338:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775025.559343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664671232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775025.559350:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559353:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775025.559354:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559357:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559360:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559363:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775025.559366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775025.559367:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9df80 x1862032664670848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775025.559376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664671360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775025.559387:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664670848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775025.559390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559400:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775025.559403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775025.559404:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775025.559412:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559417:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775025.559422:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775025.559423:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559431:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664670848:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775025.559431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775025.559449:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664671488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775025.559456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775025.559461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775025.559463:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670976:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775025.559464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775025.559474:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16793 00000100:00080000:3.0:1775775025.559479:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775025.559483:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775025.559489:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775025.559499:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775025.559504:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664671616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775025.559516:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9e680 x1862032664670976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559532:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664670976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559541:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559549:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664670976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775025.559563:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775025.559586:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9ea00 x1862032664671104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664671104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559623:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775025.559627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775025.559643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9c000 x1862032664671232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664671232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775025.559669:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775025.559679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9d180 x1862032664671360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664671360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775025.559705:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664671488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775025.559714:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9f800 x1862032664671488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664671488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664671488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775025.559768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664671616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775025.559789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775025.559801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775025/real 1775775025] req@ffff982980f9c700 x1862032664671616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775025.559815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664671616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775025.559823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775025.559826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775025.559829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664671616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775026.071133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775026.327179:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775027.095198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775027.351236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775028.119140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775028.375230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775029.143164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775029.399229:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775030.167182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775030.423225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775030.679162:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679174:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775030.679179:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679189:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679202:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679209:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775030.679244:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664671744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679276:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679279:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775030.679281:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679285:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679293:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679298:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679309:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775030.679322:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664671744:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775030.679327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664671872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679339:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679341:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775030.679343:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679346:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679354:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775030.679381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664672000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775030.679398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679402:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679408:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679412:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679419:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775030.679428:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775030.679436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664672128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679455:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775030.679457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679462:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679467:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679472:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679479:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775030.679483:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775030.679507:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664672256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679521:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775030.679522:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672128:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775775030.679526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775030.679531:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679536:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679541:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679546:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679552:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775030.679565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775030.679582:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664672384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775030.679596:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775030.679601:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775030.679601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664672384:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775030.679603:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775030.679612:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16798 00000100:00080000:0.0:1775775030.679617:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775030.679627:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775030.679632:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775030.679641:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664672512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775030.679668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664672512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775031.191172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775031.447245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775032.215204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775032.471240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775033.239158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775033.495241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775775033.751357:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775775033.751397:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775775033.751422:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751473:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751483:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751491:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751499:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751507:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775033.751515:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775033.751623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2d180 x1862032664672512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664672512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751674:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664672512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775033.751690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2f480 x1862032664672384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664672384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664672384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775033.751740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2d500 x1862032664672256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664672256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775033.751778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2ca80 x1862032664672128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664672128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775033.751814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2fb80 x1862032664671872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664671872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751834:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664671872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775033.751880:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2f100 x1862032664672000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664672000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751902:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751905:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751908:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775033.751918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775030/real 1775775033] req@ffff982974b2df80 x1862032664671744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775033.751930:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664671744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775033.751938:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775033.751942:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775033.751945:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664671744:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775034.263175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775034.519253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775035.287181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775035.543251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775035.799185:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799195:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775035.799200:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775035.799211:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799229:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775035.799236:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775035.799269:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664672640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775035.799309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799313:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775035.799315:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775035.799321:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799325:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775035.799331:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775035.799341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775035.799356:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664672768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775035.799360:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664672640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775035.799376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775035.799383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775035.799388:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799393:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775035.799398:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775035.799404:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775035.799416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664672896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775035.799428:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775035.799434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775035.799439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775035.799442:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799449:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775035.799453:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775035.799460:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775035.799475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664673024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775035.799481:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9d880 x1862032664672640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775035.799491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775035.799504:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775035.799510:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799514:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775035.799516:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664672640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775035.799521:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775035.799532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775035.799534:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775035.799540:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.799545:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664672640:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775035.799552:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664673152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775035.799564:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775035.799577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672768:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775035.799585:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775035.799588:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775035.799594:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799600:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775035.799605:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775035.799610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775775035.799615:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775035.799630:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664673280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775035.799643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9e300 x1862032664672768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775035.799659:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775035.799664:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775035.799667:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775035.799672:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664672768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775035.799674:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16803 00000100:00080000:3.0:1775775035.799681:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775035.799689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775035.799689:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775035.799699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775775035.799700:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775035.799706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775035.799715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775035.799718:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664673408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775035.799744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775035.799759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9ce00 x1862032664672896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775035.799778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664672896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775035.799787:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775035.799790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.799793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664672896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775035.799816:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775035.799840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775035.799852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9c700 x1862032664673024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775035.799865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664673024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775035.799873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775035.799877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.799880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775035.799887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775035.799908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775035.799919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9f800 x1862032664673152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775035.799932:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664673152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775035.799947:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775035.799951:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.799954:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775035.799973:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664673280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775035.799997:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775035.800011:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9d180 x1862032664673280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775035.800022:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664673280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775035.800030:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775035.800033:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.800036:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664673280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775035.800042:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664673408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775035.800088:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775035.800101:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775035/real 1775775035] req@ffff982980f9c000 x1862032664673408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775035.800113:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664673408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775035.800122:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775035.800125:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775035.800130:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664673408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775036.311165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775036.567247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775037.335149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775037.591245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775038.359164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775038.615211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775039.383163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775039.639219:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775040.407203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775040.663244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775040.919171:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775040.919179:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775040.919184:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919193:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00080000:3.0:1775775040.919205:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775040.919244:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664673536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775040.919270:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775040.919274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775040.919276:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919281:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00080000:3.0:1775775040.919285:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919290:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775040.919299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775040.919315:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664673664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775040.919318:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664673536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775040.919336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775040.919340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775040.919342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919348:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00080000:3.0:1775775040.919353:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919358:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775040.919365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775040.919380:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775040.919393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664673792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775040.919406:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775040.919410:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9ea00 x1862032664673536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775040.919412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775040.919416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919426:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00080000:3.0:1775775040.919432:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775040.919433:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664673536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775040.919441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775040.919449:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775040.919451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775040.919455:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919460:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664673536:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775040.919471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664673920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775040.919481:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673664:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775040.919500:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775040.919506:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775040.919506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775040.919511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919517:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00100000:0.0:1775775040.919519:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9e680 x1862032664673664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775040.919525:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919531:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919540:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664673664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775040.919540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775040.919550:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919553:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919559:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775040.919560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664674048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775040.919567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673792:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775040.919582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775040.919586:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1775775040.919591:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775040.919591:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775040.919599:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00100000:0.0:1775775040.919602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9df80 x1862032664673792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775040.919605:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919614:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664673792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775040.919622:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775040.919626:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775040.919641:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775040.919646:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664674176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775040.919657:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775040.919659:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775040.919664:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775040.919667:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775040.919672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9c380 x1862032664673920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775040.919674:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16808 00000100:00080000:3.0:1775775040.919682:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775040.919689:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664673920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775040.919696:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775040.919724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664673920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775040.919741:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664674304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775040.919742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775040.919776:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9f100 x1862032664674048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775040.919828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664674048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775040.919837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775040.919859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664674176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775040.919880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9f480 x1862032664674176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775040.919904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664674176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775040.919922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664674176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775040.919936:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664674304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775040.919959:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775040.919970:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775040/real 1775775040] req@ffff982980f9dc00 x1862032664674304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775040.919983:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664674304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775040.919992:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775040.919995:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775040.919998:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664674304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775041.431133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775041.687217:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775042.455168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775042.711239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775043.479467:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775043.735213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775044.503181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775044.759232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775045.527154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775045.783197:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775046.039177:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039185:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775046.039189:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039198:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039208:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039214:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775046.039247:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664674432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775046.039272:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039275:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775046.039277:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039280:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039289:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039292:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039302:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775046.039312:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664674560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775046.039317:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664674432:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775046.039325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039328:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775046.039329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039337:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039340:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775046.039348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664674688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775046.039355:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039357:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775046.039358:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039361:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039363:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039365:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775046.039380:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664674816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775046.039382:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775046.039390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039393:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775046.039395:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039399:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039401:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775046.039412:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664674944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775046.039419:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775046.039422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775046.039425:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039428:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775046.039431:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2c380 x1862032664674432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775046.039433:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775046.039450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664675072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775046.039457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775046.039460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775046.039461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775046.039465:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664674432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775046.039467:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16813 00000100:00080000:0.0:1775775046.039471:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775046.039477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775046.039484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775046.039491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775046.039493:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664675200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775046.039502:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039509:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664674432:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775046.039535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775046.039562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039577:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2e680 x1862032664674560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664674560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039613:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775046.039619:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775046.039659:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2e300 x1862032664674688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664674688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039699:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775046.039705:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775046.039726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2df80 x1862032664674816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664674816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775046.039768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775046.039787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2f100 x1862032664674944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664674944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664674944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775046.039832:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775046.039850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039860:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2fb80 x1862032664675072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664675072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775046.039885:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775046.039899:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775046.039906:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775046/real 1775775046] req@ffff982974b2ca80 x1862032664675200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775046.039913:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664675200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775046.039918:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775046.039920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775046.039922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775046.551192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775046.807266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775047.575193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775047.831199:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775048.599169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775048.855241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775049.623175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775049.879212:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775050.647180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775050.903266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775051.159237:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159248:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775051.159253:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159264:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159278:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159285:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775051.159333:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664675328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775051.159365:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159368:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775051.159369:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159373:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159375:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159378:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159388:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775051.159400:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664675456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775051.159416:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664675328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775051.159417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775051.159426:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159430:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159432:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159435:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775051.159448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664675584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775051.159455:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159458:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775051.159459:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159462:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159466:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159469:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159472:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775051.159480:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664675712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775051.159486:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159489:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775051.159492:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775051.159493:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775051.159498:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159500:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159503:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159506:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775051.159514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664675840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775051.159520:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159523:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775051.159525:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159527:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159530:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775051.159545:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159548:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775051.159554:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe3480 x1862032664675328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775051.159558:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664675968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775051.159592:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775051.159594:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775051.159596:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775051.159598:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16818 00000100:00080000:1.0:1775775051.159601:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775051.159605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664675328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775051.159606:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775051.159612:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775051.159623:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664676096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775051.159625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664675328:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775051.159663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775051.159688:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159702:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe2a00 x1862032664675456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664675456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775051.159746:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775051.159766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe2680 x1862032664675584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664675584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159792:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775051.159798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775051.159811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159817:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe2300 x1862032664675712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664675712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775051.159838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775051.159848:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe1880 x1862032664675840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159860:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664675840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159869:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664675840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775051.159873:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775051.159883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe0000 x1862032664675968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159896:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664675968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664675968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775051.159907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775051.159916:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775051.159922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775051/real 1775775051] req@ffff982981fe0380 x1862032664676096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775051.159930:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664676096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775051.159935:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775051.159936:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775051.159938:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775051.671189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775051.927305:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775052.695191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775052.951306:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775053.719283:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775053.975244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775054.743193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775054.999242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775055.767222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775056.023268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775056.279159:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279168:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775056.279172:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775056.279181:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00080000:0.0:1775775056.279193:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279199:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775056.279232:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664676224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775056.279257:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279260:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775056.279262:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775056.279265:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00080000:0.0:1775775056.279268:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279272:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775056.279284:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775056.279299:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664676352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775056.279311:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279316:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775056.279316:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664676224:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775056.279321:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775056.279327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00080000:0.0:1775775056.279331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279336:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775056.279342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775056.279357:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664676480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775056.279367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775056.279372:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775056.279377:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775056.279380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00080000:0.0:1775775056.279386:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279391:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775056.279398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775056.279409:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2d500 x1862032664676224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775056.279414:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664676608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775056.279434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279438:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775056.279440:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775056.279459:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664676224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775056.279468:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279472:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775056.279476:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00100000:3.0:1775775056.279477:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664676224:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775056.279483:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775056.279495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775056.279501:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775056.279509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664676736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775056.279523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775056.279524:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775056.279530:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775056.279532:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775056.279538:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2f480 x1862032664676352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775056.279539:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00080000:0.0:1775775056.279547:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775056.279554:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664676352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775056.279555:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775056.279564:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775056.279575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279579:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775056.279584:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664676864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775056.279592:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775056.279606:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775056.279610:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775775056.279613:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775056.279614:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775056.279621:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16823 00000100:00100000:3.0:1775775056.279624:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2d180 x1862032664676480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775056.279627:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775056.279635:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775056.279640:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664676480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775056.279641:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775056.279655:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664676992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775056.279656:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775056.279680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775056.279695:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775056.279703:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2ca80 x1862032664676608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775056.279711:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664676608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775056.279716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775056.279732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775056.279742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775056.279749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2fb80 x1862032664676736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775056.279756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664676736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775056.279761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279765:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664676736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775056.279768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775056.279777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775056.279782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2f100 x1862032664676864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775056.279789:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664676864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775056.279794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775056.279801:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775056.279810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775056.279816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775056/real 1775775056] req@ffff982974b2df80 x1862032664676992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775056.279823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664676992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775056.279827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775056.279829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775056.279831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664676992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775056.791303:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775057.047295:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775057.815214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775058.071302:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775058.839202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775059.095299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775059.863193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775060.119299:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775060.887209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775061.143254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775061.399197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775061.399212:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399223:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399237:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775061.399289:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664677120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775061.399318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775061.399324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399333:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399338:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775061.399350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775061.399383:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664677248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775061.399397:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664677120:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775061.399424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775061.399429:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399433:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399437:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399441:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775061.399448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775061.399460:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664677376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775061.399472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399476:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775061.399478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775061.399483:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775061.399488:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399494:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399499:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775061.399506:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775061.399524:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664677504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775061.399532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b5f80 x1862032664677120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775061.399542:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399562:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775061.399569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399576:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00100000:3.0:1775775061.399582:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664677120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775061.399586:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775061.399599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775061.399600:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775061.399610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775775061.399614:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775061.399614:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664677120:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775061.399638:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664677632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775061.399649:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775061.399651:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775775061.399657:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677248:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775061.399658:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399671:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399676:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399680:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775061.399684:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775061.399695:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775061.399696:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664677760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775061.399709:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775061.399715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b4700 x1862032664677248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775061.399717:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775061.399722:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775061.399729:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16828 00000100:00080000:2.0:1775775061.399732:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775061.399734:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775061.399737:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775061.399756:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664677888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775061.399758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664677248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.399823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.399826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.399828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775061.399833:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775061.399848:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775061.399855:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b5c00 x1862032664677376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775061.399862:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664677376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.399867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.399869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.399871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775061.399875:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775061.399887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775061.399892:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b5180 x1862032664677504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775061.399899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664677504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.399904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.399906:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.399909:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775061.399913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775061.399924:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775061.399929:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b4a80 x1862032664677632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775061.399935:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664677632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.399940:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.399942:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.399944:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664677632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775061.399957:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664677760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775061.399967:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775061.399973:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b6d80 x1862032664677760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775061.399980:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664677760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.399985:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.399987:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.399989:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664677760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775061.400057:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664677888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775061.400105:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775061.400118:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775061/real 1775775061] req@ffff9829764b5f80 x1862032664677888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775061.400127:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664677888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775061.400132:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775061.400135:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775061.400137:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664677888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775061.911170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775062.167247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775062.935149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775063.191246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775063.959202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775064.215232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775064.983134:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775065.239221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775066.007181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775066.263312:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775066.519210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519218:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775066.519222:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519232:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519243:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519248:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775066.519301:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664678016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519327:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519330:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775066.519332:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519335:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519338:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519341:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775066.519350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775066.519363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664678144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775066.519382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519386:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519391:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775066.519402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775066.519414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664678016:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775066.519422:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664678272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519440:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775066.519442:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519448:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519453:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519457:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775066.519465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775066.519480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664678400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775066.519497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519503:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519509:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519513:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775066.519519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775066.519522:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775066.519538:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664678528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519552:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519556:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775066.519558:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775066.519564:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519569:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519573:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775066.519579:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775066.519583:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9ed80 x1862032664678016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775066.519595:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664678656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775066.519616:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775066.519620:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775066.519623:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775066.519629:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664678016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775066.519633:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16834 00000100:00080000:3.0:1775775066.519641:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775066.519651:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775066.519656:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775066.519657:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775066.519669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775066.519677:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664678784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775066.519678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664678016:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775066.519701:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775066.519752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.519765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9fb80 x1862032664678144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.519781:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664678144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.519788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.519790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.519793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775066.519819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775066.519833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.519847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9d500 x1862032664678272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.519855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664678272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.519861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.519864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.519866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775066.519870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775066.519883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.519889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9dc00 x1862032664678400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.519896:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664678400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.519909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.519913:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.519916:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775066.519923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775066.519944:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.519956:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9f480 x1862032664678528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.519969:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664678528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.519976:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.519979:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.519981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664678528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775066.519986:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664678656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775066.520003:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.520010:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9f100 x1862032664678656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.520017:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664678656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.520023:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.520025:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.520028:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664678656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775066.520032:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664678784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775066.520048:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775066.520058:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775066/real 1775775066] req@ffff982980f9c380 x1862032664678784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775066.520108:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664678784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775066.520114:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775066.520116:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775066.520119:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664678784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775067.031281:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775067.287230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775068.055178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775068.311245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775069.079173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775069.335310:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775070.103158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775070.359231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775071.127201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775071.383320:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775071.639194:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775071.639213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639227:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639243:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775071.639313:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664678912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775071.639352:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775071.639359:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639364:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639373:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775071.639386:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775071.639402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664679040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775071.639414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664678912:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775071.639417:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775071.639426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639432:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639436:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639440:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775071.639447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775071.639461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664679168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775071.639473:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639477:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775071.639479:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639484:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639488:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639493:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775071.639500:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775071.639514:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775071.639515:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664679296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775071.639527:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639531:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775071.639533:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639538:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639546:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775071.639552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775071.639564:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664679424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775071.639566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9df80 x1862032664678912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775071.639598:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775071.639604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664678912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775071.639604:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775071.639609:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639621:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:0.0:1775775071.639624:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775071.639626:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775071.639630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775071.639634:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664678912:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775071.639640:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775071.639654:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664679552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775071.639661:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775071.639665:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775071.639667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679040:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775071.639667:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775071.639676:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16839 00000100:00080000:3.0:1775775071.639678:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775071.639681:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775071.639685:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775071.639694:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664679680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775071.639697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639711:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9e680 x1862032664679040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664679040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639732:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775071.639736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775071.639748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9ea00 x1862032664679168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664679168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775071.639774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775071.639785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9c000 x1862032664679296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664679296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775071.639811:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775071.639822:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9d180 x1862032664679424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664679424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775071.639846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664679552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775071.639856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9f800 x1862032664679552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664679552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639876:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664679552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775071.639891:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664679680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775071.639901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775071.639918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775071/real 1775775071] req@ffff982980f9c700 x1862032664679680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775071.639926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664679680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775071.639931:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775071.639933:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775071.639935:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664679680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775072.151212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775072.407267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775073.175212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775073.431307:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775074.199179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775074.455256:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775075.223275:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775075.479281:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775076.247183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775076.503270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775076.759203:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759214:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775076.759219:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759250:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775076.759290:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664679808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775076.759320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759325:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775076.759326:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759331:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759336:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759340:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775076.759351:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775076.759365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664679936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775076.759376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759380:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775076.759382:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759388:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00100000:0.0:1775775076.759393:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664679808:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775076.759399:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759405:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775076.759410:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775076.759424:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664680064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775076.759435:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775076.759441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759445:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759450:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759454:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775076.759460:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775076.759467:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775076.759472:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664680192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775076.759484:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759488:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775076.759490:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759495:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759500:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759505:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775076.759510:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe3b80 x1862032664679808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775076.759512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775076.759526:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664680320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775076.759537:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775076.759541:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775076.759543:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759548:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759552:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775076.759553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664679808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775076.759559:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775076.759567:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775076.759574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775775076.759581:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664680448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775076.759582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.759592:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664679808:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775076.759601:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775076.759607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679936:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775076.759607:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775076.759611:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775076.759619:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16844 00000100:00080000:1.0:1775775076.759624:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775076.759635:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775076.759641:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775076.759647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775076.759659:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664680576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775076.759677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe1f80 x1862032664679936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.759694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664679936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.759703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.759707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.759711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664679936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775076.759735:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775076.759759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775076.759772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe0a80 x1862032664680064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.759785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664680064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.759794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.759798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.759802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775076.759809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775076.759833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775076.759844:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe0e00 x1862032664680192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.759857:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664680192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.759866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.759869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.759873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775076.759879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775076.759900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775076.759912:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe1c00 x1862032664680320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.759926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664680320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.759934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.759938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.759941:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775076.759947:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664680448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775076.759968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775076.759980:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe1500 x1862032664680448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.759993:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664680448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.760001:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.760004:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.760008:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664680448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775076.760014:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664680576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775076.760036:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775076.760047:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775076/real 1775775076] req@ffff982981fe1180 x1862032664680576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775076.760060:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664680576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775076.760129:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775076.760135:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775076.760139:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664680576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775077.271175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775077.527288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775078.295185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775078.551246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775079.319217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775079.575257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775080.343172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775080.599310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775081.367192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775081.623308:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775081.879218:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879240:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775081.879249:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879261:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879276:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775081.879284:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775081.879337:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664680704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775081.879373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879377:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775081.879379:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879384:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879388:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775081.879394:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775081.879419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664680832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775081.879429:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879433:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775081.879436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879440:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879444:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775081.879448:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775081.879456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664680704:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775081.879468:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664680960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775081.879479:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879483:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775081.879485:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879489:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879493:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775081.879498:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879504:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775081.879516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664681088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775081.879526:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879531:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775081.879533:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879537:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879542:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775081.879544:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775081.879549:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879555:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775081.879570:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664681216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775081.879580:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775081.879583:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775081.879585:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775081.879590:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00100000:0.0:1775775081.879593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b5500 x1862032664680704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775081.879595:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775081.879604:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879614:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775081.879623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664680704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775081.879629:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664681344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775081.879646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775081.879648:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775081.879654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775081.879654:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775081.879659:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775081.879662:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664680704:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775081.879667:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16849 00000100:00080000:2.0:1775775081.879673:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775081.879675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680832:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775081.879680:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775081.879688:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775081.879702:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664681472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775081.879707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.879738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b6a00 x1862032664680832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.879754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664680832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.879763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.879766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.879769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775081.879799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775081.879820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.879831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b7100 x1862032664680960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.879843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664680960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.879858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.879861:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.879865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664680960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775081.879870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775081.879890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.879900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b7b80 x1862032664681088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.879911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664681088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.879919:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.879922:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.879925:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775081.879931:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775081.879951:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.879961:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b4e00 x1862032664681216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.879972:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664681216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.879980:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.879983:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.879986:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775081.879992:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664681344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775081.880026:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.880036:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b6680 x1862032664681344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.880048:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664681344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.880055:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.880058:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.880061:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664681344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775081.880123:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664681472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775081.880144:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775081.880155:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775081/real 1775775081] req@ffff9829764b4000 x1862032664681472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775081.880166:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664681472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775081.880173:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775081.880176:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775081.880179:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664681472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775082.391158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775082.647257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775083.415159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775083.671270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775084.439144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775084.695218:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775085.463237:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775085.719264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775086.487186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775086.743240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775086.999192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999204:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775086.999210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999221:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999235:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775086.999281:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664681600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775086.999307:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775086.999312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999325:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775086.999335:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775086.999350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664681728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775086.999353:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664681600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775086.999360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775086.999365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999368:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999371:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999374:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775086.999379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775086.999387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664681856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775086.999395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775086.999399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999408:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775086.999412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775086.999421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664681984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775086.999424:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775086.999429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775086.999434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999437:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999440:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999443:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775086.999447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775086.999454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664682112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775086.999460:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9ce00 x1862032664681600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775086.999462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775086.999466:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775086.999468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999470:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999473:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999476:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999478:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664681600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775086.999480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775086.999487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664682240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775086.999489:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999493:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775086.999495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775086.999496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664681600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775086.999499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775086.999500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775086.999503:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16854 00000100:00080000:3.0:1775775086.999505:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775086.999508:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775086.999512:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775086.999519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664682368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775086.999523:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775086.999543:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9e300 x1862032664681728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999566:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664681728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999577:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775086.999586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775086.999608:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999615:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9d880 x1862032664681856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664681856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999628:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999629:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999631:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775086.999634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775086.999647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999652:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9c700 x1862032664681984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664681984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664681984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775086.999669:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775086.999678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9f800 x1862032664682112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664682112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775086.999704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664682240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775086.999713:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9d180 x1862032664682240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999724:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664682240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999731:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664682240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775086.999742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664682368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775086.999751:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775086.999756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775086/real 1775775086] req@ffff982980f9c000 x1862032664682368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775086.999762:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664682368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775086.999770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775086.999773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775086.999774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664682368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775087.511173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775087.767246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775088.535185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775088.791256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775089.559185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775089.815254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775090.583191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775090.839258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775091.607188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775091.863254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775092.119102:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119112:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775092.119116:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775092.119129:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119144:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119152:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775092.119196:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664682496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775092.119218:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775092.119222:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775092.119226:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119229:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119234:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119246:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775092.119262:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664682624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775092.119263:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664682496:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775092.119277:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119280:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775092.119281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775092.119293:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119297:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119300:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119304:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775092.119315:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664682752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775092.119324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119328:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775092.119330:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775092.119335:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775092.119336:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119342:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119347:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775092.119367:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664682880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775092.119377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775092.119381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775092.119385:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9ea00 x1862032664682496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775092.119387:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119393:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119400:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775092.119410:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664682496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775092.119420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664683008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775092.119421:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119431:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775092.119435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119439:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775092.119441:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775092.119447:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00100000:0.0:1775775092.119448:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664682496:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775092.119451:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119459:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775092.119476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664683136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775092.119480:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682624:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775092.119494:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775092.119497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775092.119499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775092.119504:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16859 00000100:00080000:3.0:1775775092.119508:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775092.119512:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775092.119516:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775775092.119520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775092.119529:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9e680 x1862032664682624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775092.119532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664683264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775092.119551:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664682624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119558:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119562:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775092.119566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775092.119579:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775092.119585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9df80 x1862032664682752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775092.119593:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664682752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119607:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775092.119613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775092.119633:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775092.119643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9c380 x1862032664682880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775092.119653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664682880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119660:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119663:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664682880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775092.119666:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775092.119680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775092.119686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9f100 x1862032664683008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775092.119692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664683008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775092.119712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664683136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775092.119722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775092.119727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9f480 x1862032664683136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775092.119733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664683136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664683136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775092.119743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664683264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775092.119752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775092.119758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775092/real 1775775092] req@ffff982980f9dc00 x1862032664683264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775092.119765:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664683264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775092.119769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775092.119771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775092.119772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664683264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775092.631193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775092.887263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775093.655211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775093.911291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775094.679202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775094.935290:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775095.703172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775095.959265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775096.727185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775096.983260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775097.239173:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239193:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775097.239199:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239226:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239234:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775097.239287:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664683392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239325:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775097.239327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239332:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239337:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239342:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775097.239354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775097.239371:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664683520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775097.239387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775097.239395:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664683392:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775097.239509:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683520:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775097.239516:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239526:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239534:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775097.239546:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775097.239576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664683648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239602:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239606:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775097.239608:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239612:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239616:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239620:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775097.239626:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775097.239637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775097.239642:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664683776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239655:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239660:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775097.239662:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239666:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239671:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239676:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775097.239681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683776:192.168.201.118@tcp:8:kworker.0 00000008:00100000:2.0:1775775097.239683:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775097.239701:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664683904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239714:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775097.239719:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683904:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775775097.239720:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775097.239724:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239746:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239751:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775097.239756:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775097.239762:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775097.239777:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664684032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775097.239796:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775097.239800:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775097.239803:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775097.239807:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16864 00000100:00080000:2.0:1775775097.239813:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775097.239819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684032:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775097.239820:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775097.239830:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775097.239846:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664684160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775097.239865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775097.751154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775098.007215:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775098.775147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775099.031267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775099.799140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775100.055240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775100.311354:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775100.311390:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775100.311405:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311441:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311450:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311459:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311468:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311476:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775100.311488:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775100.311511:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b6a00 x1862032664684032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311541:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664684032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311554:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311564:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775100.311572:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b7100 x1862032664683904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311588:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664683904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775100.311610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b7b80 x1862032664683776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664683776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311637:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775100.311644:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b4e00 x1862032664683648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664683648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775100.311709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b6680 x1862032664683520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664683520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664683520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775100.311746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b4000 x1862032664683392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664683392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664683392:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775100.311810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775097/real 1775775100] req@ffff9829764b5500 x1862032664684160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775100.311822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664684160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775100.311829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775100.311833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775100.311835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775100.823181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775101.079252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775101.847154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775102.103236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775102.359213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359221:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775102.359225:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359233:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775102.359248:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775102.359278:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664684288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775102.359309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359312:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775102.359314:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359319:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359323:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775102.359328:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359338:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775102.359353:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664684416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775102.359364:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359367:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775102.359369:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359372:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775102.359380:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359385:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775102.359397:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664684544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775102.359407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775102.359413:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00100000:0.0:1775775102.359421:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664684288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775102.359424:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775102.359432:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359440:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775102.359455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664684672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775102.359467:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775102.359473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359477:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359483:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775102.359487:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359493:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775102.359505:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775102.359508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664684800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775102.359521:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775102.359525:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775102.359534:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775102.359539:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359545:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775102.359550:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe1180 x1862032664684288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775102.359553:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359564:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775102.359578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664684288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775102.359583:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664684928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775102.359602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775102.359605:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775102.359610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775102.359612:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775102.359616:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775102.359619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664684288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775102.359624:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16869 00000100:00080000:1.0:1775775102.359631:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775102.359634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684416:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775102.359652:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775102.359660:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775102.359676:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664685056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775102.359687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.359707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe1500 x1862032664684416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.359729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664684416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.359739:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.359743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.359747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775102.359753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775102.359777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.359791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe1c00 x1862032664684544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.359803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664684544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.359810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.359813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.359817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775102.359845:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775102.359868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.359882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe0e00 x1862032664684672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.359893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664684672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.359908:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.359911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.359914:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775102.359921:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775102.359941:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.359952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe0a80 x1862032664684800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.359963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664684800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.359971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.359974:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.359977:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664684800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775102.359983:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775102.360003:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.360013:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe1f80 x1862032664684928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.360024:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664684928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.360032:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.360035:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.360102:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664684928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775102.360109:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775102.360132:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775102.360144:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775102/real 1775775102] req@ffff982981fe3b80 x1862032664685056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775102.360156:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664685056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775102.360163:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775102.360167:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775102.360170:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775102.871168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775103.127252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775103.895305:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775104.151241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775104.919188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775105.175259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775105.943214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775106.199225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775106.967192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775107.223260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775107.479196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479207:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775107.479212:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479222:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00080000:1.0:1775775107.479235:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479241:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775107.479286:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664685184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775107.479309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479314:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775107.479315:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479320:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00080000:1.0:1775775107.479324:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479329:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479339:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775107.479352:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664685312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775107.479361:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479365:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775107.479367:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479370:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00100000:0.0:1775775107.479372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664685184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775107.479374:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479378:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775107.479391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664685440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775107.479399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775107.479404:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479407:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00080000:1.0:1775775107.479421:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479425:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479430:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775107.479435:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775107.479439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664685568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775107.479456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775107.479461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479465:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00100000:0.0:1775775107.479468:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe0380 x1862032664685184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775107.479468:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479472:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479477:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775107.479486:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664685184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775107.479486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664685696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775107.479496:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775107.479498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775107.479500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775107.479501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775107.479502:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775107.479505:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00100000:0.0:1775775107.479507:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664685184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775107.479508:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479512:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775107.479523:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664685824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775107.479528:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685312:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775107.479532:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775107.479536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775107.479537:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775107.479541:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16874 00000100:00080000:1.0:1775775107.479546:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775107.479563:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775107.479564:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775775107.479569:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775107.479578:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe0000 x1862032664685312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775107.479579:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664685952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775107.479591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664685312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479600:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479603:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479606:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775107.479612:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775107.479632:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775107.479643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe1880 x1862032664685440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775107.479653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664685440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479661:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775107.479674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775107.479697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775107.479709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe2300 x1862032664685568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775107.479722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664685568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775107.479742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775107.479764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775107.479774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe2680 x1862032664685696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775107.479787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664685696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664685696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775107.479819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775107.479839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775107.479849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe2a00 x1862032664685824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775107.479860:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664685824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479871:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775107.479879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775107.479895:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775107.479904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775107/real 1775775107] req@ffff982981fe3480 x1862032664685952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775107.479915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664685952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775107.479922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775107.479926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775107.479929:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664685952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775107.991155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775108.247213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775109.015167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775109.271278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775110.039203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775110.295282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775111.063159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775111.319254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775112.087231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775112.343244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775112.599150:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599165:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775112.599169:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775112.599177:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599188:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599194:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775112.599225:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664686080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775112.599249:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599253:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775112.599254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775112.599260:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599262:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599265:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599273:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775112.599282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664686208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775112.599295:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599298:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775112.599301:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775112.599303:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664686080:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775112.599306:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599312:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599315:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599318:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775112.599326:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664686336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775112.599333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599335:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775112.599337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775112.599340:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599343:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599345:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599349:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775112.599356:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664686464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775112.599362:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599365:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775112.599367:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775112.599370:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775112.599374:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599377:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599379:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775112.599391:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664686592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775112.599398:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775112.599402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775112.599406:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2e300 x1862032664686080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775112.599408:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599412:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775112.599419:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775112.599431:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664686080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775112.599433:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664686720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775112.599457:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775112.599470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775112.599475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775112.599477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775112.599482:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775112.599483:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16880 00000100:00080000:0.0:1775775112.599489:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775112.599492:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664686080:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775112.599493:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775112.599499:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775112.599508:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664686848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775112.599522:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775112.599544:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599553:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2e680 x1862032664686208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599561:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664686208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599569:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599571:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775112.599577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775112.599588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2c380 x1862032664686336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599600:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664686336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775112.599613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775112.599623:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599629:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2df80 x1862032664686464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599636:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664686464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599642:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775112.599647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775112.599656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2f100 x1862032664686592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664686592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599677:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664686592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775112.599680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664686720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775112.599689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2fb80 x1862032664686720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664686720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599705:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664686720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775112.599712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664686848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775112.599721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775112.599726:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775112/real 1775775112] req@ffff982974b2ca80 x1862032664686848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775112.599732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664686848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775112.599738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775112.599740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775112.599741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664686848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775113.111179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775113.367228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775114.135176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775114.391254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775115.159213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775115.415247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775116.183199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775116.439224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775117.207148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775117.463244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775117.719211:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719223:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775117.719228:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719239:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719252:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719259:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775117.719313:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664686976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775117.719340:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775117.719348:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719352:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719357:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719363:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719388:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775117.719394:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664686976:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775117.719403:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664687104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775117.719414:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775117.719419:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719423:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719426:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719430:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775117.719444:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664687232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775117.719453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775117.719457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719461:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719465:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719468:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775117.719478:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775117.719484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664687360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775117.719492:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719496:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775117.719497:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719501:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719504:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719508:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775117.719521:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9d500 x1862032664686976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775117.719521:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664687488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775117.719533:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775117.719538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775117.719542:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664686976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775117.719542:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719545:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719549:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719553:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775117.719562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664687616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775117.719563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664686976:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775117.719572:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775117.719576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775117.719577:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775117.719581:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16885 00000100:00080000:3.0:1775775117.719584:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775117.719587:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775117.719592:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775117.719600:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664687744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775117.719731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775117.719793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.719815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9fb80 x1862032664687104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.719828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664687104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.719837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775117.719848:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775117.719860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.719866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9ed80 x1862032664687232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.719873:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664687232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.719879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775117.719887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775117.719898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.719905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9dc00 x1862032664687360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.719912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664687360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.719917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775117.719925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775117.719935:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.719941:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9f480 x1862032664687488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.719948:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664687488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.719953:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719955:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719957:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664687488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775117.719960:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664687616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775117.719969:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.719975:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9f100 x1862032664687616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.719981:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664687616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.719987:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.719989:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.719991:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664687616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775117.719995:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664687744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775117.720004:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775117.720009:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775117/real 1775775117] req@ffff982980f9c380 x1862032664687744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775117.720016:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664687744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775117.720021:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775117.720023:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775117.720025:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664687744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775118.231226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775118.487259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775119.255149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775119.511223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775120.279166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775120.535260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775121.303151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775121.559253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775122.327187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775122.583247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775122.839180:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839188:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775122.839193:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839201:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839213:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839219:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775122.839252:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664687872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775122.839276:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839309:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775122.839312:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839319:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839324:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839330:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839339:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664687872:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775122.839341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775122.839367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664688000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775122.839379:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775122.839384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839387:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839393:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775122.839397:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775122.839404:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775122.839408:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664688128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775122.839421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839424:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775122.839425:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839428:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839431:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839433:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775122.839438:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775122.839448:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9df80 x1862032664687872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775122.839448:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664688256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775122.839473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839477:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775122.839479:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839482:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839485:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839487:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775122.839491:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775122.839492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664687872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775122.839500:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664688384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775122.839515:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775122.839515:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839520:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775122.839523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775122.839526:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775122.839528:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00080000:3.0:1775775122.839533:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775122.839537:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664687872:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775122.839540:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775122.839546:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775122.839553:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664688512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775122.839559:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775122.839561:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775122.839562:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775122.839565:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16890 00000100:00100000:0.0:1775775122.839568:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688000:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775122.839569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775122.839575:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775122.839579:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775122.839587:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664688640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775122.839600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9e680 x1862032664688000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664688000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839628:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839633:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775122.839637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775122.839648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9ea00 x1862032664688128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664688128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775122.839675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775122.839686:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839692:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9c000 x1862032664688256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664688256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839704:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839706:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839708:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775122.839712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775122.839721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9d180 x1862032664688384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839734:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664688384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775122.839745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664688512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775122.839754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9f800 x1862032664688512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664688512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664688512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775122.839787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664688640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775122.839796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775122.839802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775122/real 1775775122] req@ffff982980f9c700 x1862032664688640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775122.839809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664688640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775122.839814:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775122.839815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775122.839817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664688640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775123.351132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775123.607206:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775124.375181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775124.631266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775125.399153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775125.655212:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775126.423184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775126.679307:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775127.447189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775127.703245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775127.959160:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959170:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775127.959175:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959185:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959211:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775127.959250:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664688768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775127.959271:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959274:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775127.959276:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959279:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959283:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959286:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959295:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775127.959305:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664688896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775127.959319:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664688768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775127.959322:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775127.959338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959343:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959348:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959360:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775127.959376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664689024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775127.959384:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959388:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775127.959390:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959393:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959396:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959399:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959404:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775127.959417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664689152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775127.959428:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775127.959431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959437:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775127.959438:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959442:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959445:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959447:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959453:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775127.959463:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664689280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775127.959471:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959474:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775127.959475:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775127.959481:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9d880 x1862032664688768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775127.959484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959493:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959508:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959513:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775127.959527:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664688768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775127.959536:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664689408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775127.959538:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959547:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959559:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664688768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775127.959560:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775127.959576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775127.959578:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775127.959584:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16895 00000100:00080000:3.0:1775775127.959588:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775127.959591:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775127.959594:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775127.959603:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664689536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775127.959610:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775127.959648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959663:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9e300 x1862032664688896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664688896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959683:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664688896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775127.959691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775127.959707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9ce00 x1862032664689024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664689024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775127.959742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775127.959754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9c700 x1862032664689152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664689152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775127.959781:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775127.959792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9f800 x1862032664689280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664689280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959814:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775127.959819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664689408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775127.959838:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9d180 x1862032664689408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664689408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664689408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775127.959877:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664689536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775127.959889:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775127.959896:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775127/real 1775775127] req@ffff982980f9c000 x1862032664689536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775127.959904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664689536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775127.959909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775127.959911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775127.959919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664689536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775128.471168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775128.727270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775129.495193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775129.751266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775130.519171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775130.775255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775131.543191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775131.799257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775132.567216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775132.823274:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775133.079222:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775133.079235:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775133.079241:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775133.079254:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079269:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079278:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775133.079340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664689664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775133.079377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775133.079381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775133.079383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775133.079389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079399:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775133.079412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775133.079414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664689664:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775133.079432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664689792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775133.079446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775133.079449:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775133.079451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775133.079456:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079466:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775133.079471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775133.079483:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664689920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775133.079493:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775133.079494:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775133.079500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775133.079502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775133.079507:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079513:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775133.079524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775133.079546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2d180 x1862032664689664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775133.079551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664690048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775133.079577:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775133.079581:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775133.079584:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775133.079585:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664689664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775133.079596:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:3.0:1775775133.079600:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775133.079602:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775133.079607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775133.079611:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664689664:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775133.079622:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775133.079639:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664690176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775133.079643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689792:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775133.079659:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775133.079664:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775133.079666:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775133.079670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775133.079677:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079681:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079686:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775133.079692:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775133.079695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2f480 x1862032664689792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775133.079705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664689792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079717:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775133.079719:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664690304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775133.079723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079730:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689792:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775133.079742:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775133.079744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689920:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775133.079747:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775133.079750:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775133.079755:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16900 00000100:00080000:0.0:1775775133.079763:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775133.079768:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775133.079775:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775133.079786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2d500 x1862032664689920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775133.079805:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664690432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775133.079813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664689920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775133.079831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664689920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775133.079839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775133.079857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079864:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2ca80 x1862032664690048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775133.079871:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664690048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079876:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775133.079878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775133.079893:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775133.079904:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2fb80 x1862032664690176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775133.079917:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664690176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775133.079924:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079926:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775133.079929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664690304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775133.079939:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2f100 x1862032664690304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775133.079952:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664690304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079957:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775133.079959:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079961:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664690304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775133.079964:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664690432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775133.079973:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775133.079979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775133/real 1775775133] req@ffff982974b2df80 x1862032664690432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775133.079986:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664690432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775133.079991:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775133.079993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775133.079995:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664690432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775133.591233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775133.847314:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775134.615166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775134.871225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775135.639163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775135.895235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775136.663157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775136.919268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775137.687176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775137.943296:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775138.199193:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775138.199206:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199214:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199225:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775138.199231:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775138.199266:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664690560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775138.199292:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199296:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775138.199300:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775138.199308:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775138.199317:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664690688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199327:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775138.199328:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199330:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199333:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775138.199335:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775138.199338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775138.199341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664690560:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775138.199347:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664690816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199355:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199364:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775138.199365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199369:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199374:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775138.199377:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775138.199381:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775138.199388:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664690944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775138.199398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199400:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199403:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775138.199407:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775138.199409:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775138.199413:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775138.199421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664691072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775138.199430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775138.199431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199434:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199437:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775138.199440:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775138.199444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775138.199444:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe3480 x1862032664690560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775138.199456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664691200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775138.199464:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775138.199466:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664690560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775138.199468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775138.199471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775138.199480:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16905 00000100:00080000:1.0:1775775138.199483:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775138.199484:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775138.199488:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775138.199492:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:1.0:1775775138.199493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775138.199497:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664690560:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775138.199503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664691328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775138.199506:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775138.199531:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199541:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe2a00 x1862032664690688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199552:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664690688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199571:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199575:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775138.199601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775138.199623:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199635:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe2680 x1862032664690816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664690816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775138.199681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775138.199704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe2300 x1862032664690944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664690944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664690944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775138.199741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775138.199755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe1880 x1862032664691072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664691072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199774:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199776:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775138.199781:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664691200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775138.199793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe0000 x1862032664691200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664691200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664691200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775138.199834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664691328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775138.199863:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775138.199875:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775138/real 1775775138] req@ffff982981fe0380 x1862032664691328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775138.199886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664691328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775138.199893:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775138.199897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775138.199901:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664691328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775138.711136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775138.967203:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775139.735161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775139.991209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775140.759169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775141.015261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775141.783173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775142.039239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775142.807180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775143.063220:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775143.319188:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319200:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775143.319205:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319216:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00080000:3.0:1775775143.319229:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319236:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775143.319271:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664691456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775143.319302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319307:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775143.319309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319313:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00080000:3.0:1775775143.319318:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319323:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775143.319349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664691584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775143.319364:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775143.319370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319379:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00100000:0.0:1775775143.319384:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664691456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775143.319385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775143.319419:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664691712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775143.319431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319434:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775143.319436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00080000:3.0:1775775143.319446:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319452:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319459:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775143.319471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664691840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775143.319482:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319486:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775143.319488:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319495:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16910 00000800:00000100:0.0:1775775143.319497:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775143.319507:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319514:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775143.319536:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664691968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775143.319547:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775143.319550:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775143.319552:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775143.319556:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9ea00 x1862032664691456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775143.319561:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00080000:3.0:1775775143.319573:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319583:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319590:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775143.319603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664691456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775143.319608:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664692096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775143.319655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775143.319667:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775143.319670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775143.319678:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775143.319682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664691456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775143.319683:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775143.319698:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16910 00000100:00080000:3.0:1775775143.319703:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775143.319709:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775143.319716:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775143.319725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775143.319734:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664692224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775143.319763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.319783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9e680 x1862032664691584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.319799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664691584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.319809:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.319814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.319818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775143.319825:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775143.319848:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.321292:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9df80 x1862032664691712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.321308:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664691712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.321317:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.321320:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.321323:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775143.321332:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775143.321372:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.321379:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9c380 x1862032664691840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.321387:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664691840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.321393:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.321395:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.321397:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775143.321400:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775143.321411:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.321416:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9f100 x1862032664691968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.321423:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664691968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.321429:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.321431:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.321433:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664691968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775143.321436:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775143.321445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.321450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9f480 x1862032664692096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.321457:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664692096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.321462:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.321464:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.321465:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775143.321488:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775143.321499:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775143.321505:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775143/real 1775775143] req@ffff982980f9dc00 x1862032664692224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775143.321516:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664692224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775143.321525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775143.321528:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775143.321532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775143.831215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775144.087292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775144.855185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775145.111231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775145.879156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775146.135216:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775146.903166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775147.159220:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775147.927169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775148.183239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775148.439160:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775148.439168:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775148.439172:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439181:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:3.0:1775775148.439192:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439197:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775148.439231:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664692352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775148.439254:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775148.439258:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775148.439259:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439262:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:3.0:1775775148.439264:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439267:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775148.439278:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775148.439294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664692480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775148.439301:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664692352:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775148.439312:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775148.439316:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775148.439318:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439324:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:3.0:1775775148.439329:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439333:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775148.439340:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775148.439352:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664692608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775148.439362:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775148.439368:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775148.439369:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775148.439372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439379:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:3.0:1775775148.439384:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439390:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775148.439396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775148.439402:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9ed80 x1862032664692352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775148.439411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664692736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775148.439422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775148.439428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775148.439430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775148.439431:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664692352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775148.439438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:0.0:1775775148.439446:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775148.439447:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775148.439454:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775148.439457:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439460:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664692352:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775148.439465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775148.439479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664692864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775148.439484:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775148.439500:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775148.439503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775148.439505:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439509:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16915 00000800:00000100:0.0:1775775148.439514:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775148.439516:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439522:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439528:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9fb80 x1862032664692480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775148.439528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775148.439541:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664692480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775148.439547:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664692992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775148.439553:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439567:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775148.439571:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775148.439575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692608:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775148.439577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775148.439581:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775148.439586:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16915 00000100:00080000:3.0:1775775148.439591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775148.439596:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775148.439602:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775775148.439606:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775148.439618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9d500 x1862032664692608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775148.439622:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664693120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775148.439659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664692608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775148.439667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775148.439679:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775148.439697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439705:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9dc00 x1862032664692736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775148.439713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664692736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775148.439718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775148.439734:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775148.439745:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9f480 x1862032664692864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775148.439758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664692864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775148.439763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439768:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664692864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775148.439771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775148.439780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9f100 x1862032664692992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775148.439793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664692992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775148.439798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664692992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775148.439804:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664693120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775148.439813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775148.439818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775148/real 1775775148] req@ffff982980f9c380 x1862032664693120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775148.439825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664693120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775148.439831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775148.439845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775148.439848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664693120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775148.951180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775149.207253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775149.975149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775150.231234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775150.999140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775151.255230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775152.023148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775152.279249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775153.047142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775153.303237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775153.559182:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559193:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775153.559199:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559209:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00080000:0.0:1775775153.559222:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559230:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775153.559268:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664693248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775153.559302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559307:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775153.559309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559314:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00080000:0.0:1775775153.559318:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559334:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775153.559339:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664693248:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775153.559364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664693376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775153.559378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775153.559384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00080000:0.0:1775775153.559394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775153.559406:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775153.559420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664693504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775153.559439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559444:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775153.559446:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559453:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00100000:3.0:1775775153.559453:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2c380 x1862032664693248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775153.559459:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559466:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775153.559482:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664693248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775153.559493:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664693632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775153.559496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559501:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559504:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664693248:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775153.559520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775153.559527:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775153.559528:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693376:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775153.559536:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00080000:0.0:1775775153.559542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559547:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559553:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775153.559559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775153.559573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664693760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775153.559580:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2e680 x1862032664693376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775153.559588:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559592:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775153.559594:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559600:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00080000:0.0:1775775153.559603:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559619:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559626:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775153.559627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664693376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775153.559644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664693888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775153.559656:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775153.559665:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775153.559671:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775153.559673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693376:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775153.559674:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775153.559684:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16921 00000100:00100000:3.0:1775775153.559688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693504:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775153.559691:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775153.559698:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775153.559705:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775153.559717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775153.559722:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664694016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775153.559732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2e300 x1862032664693504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775153.559776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664693504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775153.559783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775153.559796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775153.559814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775153.559821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2df80 x1862032664693632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775153.559829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664693632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775153.559834:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559838:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775153.559850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775153.559861:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775153.559867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2f100 x1862032664693760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775153.559874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664693760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775153.559879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664693760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775153.559886:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664693888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775153.559895:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775153.559901:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2fb80 x1862032664693888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775153.559909:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664693888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775153.559914:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559917:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664693888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775153.559921:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775153.559930:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775153.559935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775153/real 1775775153] req@ffff982974b2ca80 x1862032664694016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775153.559942:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664694016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775153.559947:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775153.559949:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775153.559951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775154.071145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775154.327208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775155.095167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775155.351230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775156.119167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775156.375222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775157.143166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775157.399244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775158.167166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775158.423234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775158.679149:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679163:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775158.679169:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679183:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679198:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775158.679206:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775158.679256:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664694144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775158.679283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679287:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775158.679288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679292:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679296:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775158.679301:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775158.679314:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775158.679332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664694272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775158.679343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664694144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775158.679347:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679352:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775158.679354:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679360:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775158.679368:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775158.679373:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775158.679391:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664694400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775158.679398:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679402:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775158.679403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679408:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679412:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775158.679416:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775158.679420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775158.679436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775158.679451:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2d500 x1862032664694144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775158.679455:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664694528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775158.679487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679491:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775158.679493:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775158.679498:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664694144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775158.679499:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679505:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775158.679512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775158.679515:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775158.679517:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775158.679524:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679528:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664694144:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775158.679531:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664694656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775158.679540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775158.679542:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775158.679544:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679547:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00080000:0.0:1775775158.679549:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775158.679552:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694272:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775158.679557:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775158.679570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664694784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775158.679578:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775158.679583:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775158.679587:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775158.679590:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775158.679598:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16926 00000100:00100000:3.0:1775775158.679599:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2f480 x1862032664694272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775158.679606:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775158.679613:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664694272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775158.679616:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775158.679627:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775158.679627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679637:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775158.679644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664694912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775158.679645:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775158.679684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2d180 x1862032664694400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775158.679704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664694400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775158.679715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775158.679724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775158.679737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2ca80 x1862032664694528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775158.679750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664694528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775158.679755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775158.679763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775158.679782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2fb80 x1862032664694656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775158.679801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664694656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775158.679807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664694656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775158.679825:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775158.679837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679844:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2f100 x1862032664694784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775158.679851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664694784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775158.679857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775158.679864:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775158.679874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775158.679879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775158/real 1775775158] req@ffff982974b2df80 x1862032664694912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775158.679886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664694912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775158.679892:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775158.679894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775158.679895:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664694912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775159.191205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775159.447245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775160.215140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775160.471230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775161.239157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775161.495259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775162.263163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775162.519236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775163.287207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775163.543283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775163.799149:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799162:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775163.799167:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799180:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799196:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799204:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775163.799248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664695040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799280:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799285:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775163.799287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799299:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799305:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799310:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775163.799322:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775163.799323:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664695040:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775163.799338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664695168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799354:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775163.799360:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799364:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799368:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799372:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775163.799377:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775163.799389:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664695296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799400:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799404:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775163.799406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799414:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799418:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775163.799424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775163.799435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664695424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799444:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775163.799450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799454:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799458:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799462:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775163.799467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775163.799481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664695552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775163.799498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799503:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799508:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775163.799515:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775163.799516:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695168:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775163.799521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775163.799535:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664695680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775163.799547:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775163.799551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775163.799553:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775163.799558:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16931 00000100:00080000:3.0:1775775163.799562:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775163.799564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695296:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775163.799581:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775163.799588:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775163.799603:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664695808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775163.799616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775163.799639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775163.799659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664695680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775163.799676:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664695808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775164.311223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775164.567230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775165.335188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775165.591295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775166.359211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775166.615185:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775166.871336:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775166.871383:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775166.871406:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871454:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871465:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871472:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871478:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871483:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775166.871487:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775166.871588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9f800 x1862032664695680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664695680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871640:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871645:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664695680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775166.871655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9c700 x1862032664695808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664695808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664695808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775166.871696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9e680 x1862032664695168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664695168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871717:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871721:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775166.871730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9ea00 x1862032664695296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664695296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775166.871759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9c000 x1862032664695424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664695424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775166.871791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9d180 x1862032664695552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664695552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664695552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775166.871824:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775163/real 1775775166] req@ffff982980f9df80 x1862032664695040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775166.871836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664695040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775166.871843:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775166.871847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775166.871850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664695040:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775167.383200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775167.639261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775168.407183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775168.663272:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775168.919211:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775168.919219:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775168.919223:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919231:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775168.919247:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775168.919276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664695936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775168.919370:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664695936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775168.919418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775168.919422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775168.919424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919429:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775168.919440:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775168.919445:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775168.919456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775168.919492:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664696064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775168.919504:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775168.919507:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b5f80 x1862032664695936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775168.919509:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775168.919512:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919521:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919529:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775168.919533:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775168.919538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775168.919539:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664695936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775168.919553:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775168.919554:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664696192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775168.919564:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.919572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664695936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775168.919572:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775168.919578:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775168.919582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919587:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919591:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775168.919595:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775168.919601:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775168.919610:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775168.919614:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664696320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775168.919625:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775168.919629:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775168.919631:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919635:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919639:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775168.919645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775168.919646:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775168.919658:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775168.919670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b6d80 x1862032664696064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775168.919671:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664696448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775168.919707:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775168.919710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664696064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775168.919712:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775168.919715:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775168.919724:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:0.0:1775775168.919730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775168.919731:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775168.919739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775168.919740:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775168.919747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696064:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775168.919749:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775168.919757:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775168.919766:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664696576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775168.919777:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775168.919782:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775168.919786:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775168.919787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775168.919795:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16936 00000100:00080000:2.0:1775775168.919801:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775168.919805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b4a80 x1862032664696192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775168.919809:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775168.919818:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775168.919832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664696192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775168.919833:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664696704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775168.919867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775168.919874:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.919878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775168.919896:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775168.919936:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775168.919951:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b5180 x1862032664696320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775168.919963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664696320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775168.919972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775168.919975:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.919979:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775168.919986:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775168.920007:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775168.920019:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b5c00 x1862032664696448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775168.920032:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664696448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775168.920040:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775168.920044:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.920048:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775168.920054:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664696576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775168.920117:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775168.920130:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b4700 x1862032664696576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775168.920141:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664696576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775168.920150:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775168.920155:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.920158:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664696576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775168.920165:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664696704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775168.920195:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775168.920212:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775168/real 1775775168] req@ffff9829764b5f80 x1862032664696704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775168.920224:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664696704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775168.920231:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775168.920234:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775168.920237:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664696704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775169.431169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775169.687275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775170.455177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775170.711245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775171.479180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775171.735231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775172.503260:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775172.759301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775173.527212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775173.783250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775174.039184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039192:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775174.039196:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039205:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039216:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039222:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775174.039254:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664696832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775174.039281:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039285:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775174.039287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039291:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039295:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039299:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775174.039323:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664696960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775174.039333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775174.039338:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039342:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00100000:3.0:1775775174.039343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664696832:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775174.039346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775174.039363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664697088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775174.039372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775174.039376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039380:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039382:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039386:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775174.039418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664697216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775174.039419:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775174.039427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775174.039432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039435:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039438:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039442:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039446:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775174.039452:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2e300 x1862032664696832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775174.039454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664697344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775174.039464:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775174.039468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775174.039470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039473:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039476:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775174.039477:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664696832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775174.039479:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039484:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775174.039488:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775174.039492:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664697472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775174.039492:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664696832:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775174.039501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775174.039504:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696960:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775174.039505:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775174.039506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775174.039510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16941 00000100:00080000:0.0:1775775174.039513:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775174.039516:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775174.039520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775174.039525:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775174.039529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664697600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775174.039537:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2e680 x1862032664696960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039548:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664696960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039563:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664696960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775174.039588:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775174.039604:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775174.039613:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2c380 x1862032664697088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664697088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039632:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775174.039638:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775174.039649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775174.039655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2df80 x1862032664697216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039662:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664697216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775174.039673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775174.039683:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775174.039689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2f100 x1862032664697344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664697344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775174.039708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664697472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775174.039717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775174.039722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2fb80 x1862032664697472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664697472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664697472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775174.039740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664697600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775174.039749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775174.039754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775174/real 1775775174] req@ffff982974b2ca80 x1862032664697600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775174.039760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664697600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775174.039765:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775174.039767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775174.039769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664697600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775174.551175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775174.807237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775175.575166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775175.831213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775176.599180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775176.855264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775177.623170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775177.879238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775178.647207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775178.903250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775179.159229:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159243:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775179.159248:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159262:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159278:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159288:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775179.159359:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664697728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775179.159394:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159397:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159401:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159404:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159413:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775179.159423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664697856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159432:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775179.159438:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775179.159439:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664697728:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775179.159443:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159448:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159451:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775179.159462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664697984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775179.159472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159475:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159477:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159479:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159483:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775179.159491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664698112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159498:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159501:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775179.159502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159504:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159508:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159510:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159513:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775179.159521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664698240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159533:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775179.159534:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775179.159536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775179.159539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159543:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00080000:3.0:1775775179.159545:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159551:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775179.159560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664698368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775179.159571:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775179.159575:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775179.159577:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775179.159582:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16946 00000100:00100000:0.0:1775775179.159595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9ce00 x1862032664697728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775179.159596:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775179.159605:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775179.159615:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775179.159629:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664698496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775179.159641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664697728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.159655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.159661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.159665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664697728:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775179.159698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775179.159726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.159741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9e300 x1862032664697856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.159754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664697856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.159763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.159767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.159771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775179.159777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775179.159801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.159813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9d880 x1862032664697984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.159826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664697984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.159841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.159845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.159848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664697984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775179.159855:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775179.159878:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.159888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9df80 x1862032664698112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.159900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664698112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.159909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.159913:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.159917:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775179.159924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775179.159946:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.159958:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9d180 x1862032664698240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.159970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664698240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.159979:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.159982:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.159986:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775179.159993:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664698368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775179.160014:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.160024:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9c000 x1862032664698368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.160037:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664698368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.160046:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.160049:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.160052:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664698368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775179.160058:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664698496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775179.160103:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775179.160114:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775179/real 1775775179] req@ffff982980f9ea00 x1862032664698496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775179.160125:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664698496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775179.160134:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775179.160137:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775179.160140:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664698496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775179.671133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775179.927201:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775180.695197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775180.951241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775181.719203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775181.975208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775182.743152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775182.999258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775183.767185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775184.023208:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775184.279142:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279149:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775184.279153:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279161:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279171:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279176:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775184.279222:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664698624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775184.279252:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279257:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775184.279259:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279264:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279268:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279273:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279282:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775184.279294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664698752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775184.279311:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775184.279313:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664698624:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775184.279321:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775184.279324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279330:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279333:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775184.279346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664698880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775184.279353:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279356:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775184.279357:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279359:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279361:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279363:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775184.279373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664699008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775184.279378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775184.279382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279384:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279393:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279395:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775184.279407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664699136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775184.279408:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775184.279416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279419:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775184.279421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775184.279424:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279428:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279430:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775184.279440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664699264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775184.279446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775184.279448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775184.279450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775184.279455:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9e680 x1862032664698624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775184.279456:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16951 00000100:00080000:3.0:1775775184.279460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775184.279466:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775184.279470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775184.279481:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664699392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775184.279491:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664698624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279505:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279510:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279514:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664698624:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775184.279540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775184.279568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9c700 x1862032664698752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279595:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664698752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279604:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279610:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775184.279617:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775184.279638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9f800 x1862032664698880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664698880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279673:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664698880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775184.279682:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775184.279703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9c380 x1862032664699008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664699008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279735:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775184.279748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775184.279770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9f100 x1862032664699136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664699136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279809:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775184.279821:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664699264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775184.279842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9f480 x1862032664699264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279866:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664699264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664699264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775184.279887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664699392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775184.279907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775184.279918:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775184/real 1775775184] req@ffff982980f9dc00 x1862032664699392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775184.279930:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664699392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775184.279939:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775184.279942:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775184.279946:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664699392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775184.791161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775185.047260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775185.815222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775186.071190:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775186.839194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775187.095284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775187.863193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775188.119218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775188.887208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775189.143262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775189.399181:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399188:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775189.399192:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399201:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399212:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399218:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775189.399250:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664699520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775189.399271:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399274:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775189.399275:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399279:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399281:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399292:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399300:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775189.399310:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664699648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775189.399315:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664699520:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775189.399321:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399324:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775189.399327:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399333:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399336:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399339:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775189.399351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664699776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775189.399358:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399361:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775189.399362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399365:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399369:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399373:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775189.399386:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775189.399393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664699904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775189.399405:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399409:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775189.399411:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775189.399417:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b5500 x1862032664699520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775189.399428:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399433:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775189.399436:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664699520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775189.399440:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775189.399458:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775189.399462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664700032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775189.399468:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775189.399476:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775189.399477:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664699520:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775189.399482:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775189.399485:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399490:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399492:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399495:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775189.399504:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664700160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775189.399510:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699648:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775189.399514:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775189.399525:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775189.399526:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775189.399530:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16956 00000100:00080000:2.0:1775775189.399534:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775189.399536:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775189.399541:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775189.399545:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775189.399554:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664700288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775189.399564:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b4000 x1862032664699648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664699648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399583:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775189.399595:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775189.399611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775189.399618:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b6680 x1862032664699776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664699776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399632:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775189.399644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775189.399665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775189.399677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b4e00 x1862032664699904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664699904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664699904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775189.399722:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775189.399739:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775189.399747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b7b80 x1862032664700032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664700032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775189.399776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664700160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775189.399786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775189.399792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b7100 x1862032664700160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664700160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664700160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775189.399812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664700288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775189.399821:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775189.399826:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775189/real 1775775189] req@ffff9829764b6a00 x1862032664700288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775189.399833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664700288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775189.399838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775189.399840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775189.399842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664700288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775189.911227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775190.167285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775190.935193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775191.191258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775191.959205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775192.215423:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775192.983199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775193.239284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775194.007161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775194.263296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775194.519183:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519194:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775194.519200:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775194.519210:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519223:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519230:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775194.519268:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664700416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775194.519297:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519301:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775194.519303:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775194.519307:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519311:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519315:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775194.519325:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775194.519340:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664700416:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775194.519340:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664700544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775194.519357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519360:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775194.519362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775194.519367:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519372:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519376:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775194.519382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775194.519394:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664700672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775194.519411:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775194.519414:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775194.519421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775194.519435:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519445:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775194.519451:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775194.519455:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b6a00 x1862032664700416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775194.519465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664700800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775194.519474:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664700416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775194.519478:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775194.519490:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775194.519492:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775194.519496:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:0.0:1775775194.519499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775194.519503:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775194.519504:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664700416:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775194.519511:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775194.519516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775194.519527:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664700928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775194.519528:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700544:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775194.519545:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775194.519548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775194.519551:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775194.519556:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775194.519557:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519563:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519568:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775194.519572:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b7100 x1862032664700544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775194.519575:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775194.519586:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664700544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775194.519594:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664701056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775194.519606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519614:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775194.519616:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775194.519618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700544:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775194.519625:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775194.519628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700672:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775194.519628:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775194.519638:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16962 00000100:00080000:2.0:1775775194.519644:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775194.519650:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775194.519651:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775194.519659:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775194.519664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b7b80 x1862032664700672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775194.519675:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664701184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775194.519681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664700672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775194.519700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775194.519710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775194.519719:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775194.519741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775194.519750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b4e00 x1862032664700800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775194.519757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664700800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775194.519763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775194.519768:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775194.519781:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775194.519792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775194.519798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b6680 x1862032664700928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775194.519805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664700928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775194.519810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775194.519815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664700928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775194.519818:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775194.519827:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775194.519832:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b4000 x1862032664701056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775194.519839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664701056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775194.519844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775194.519848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775194.519851:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775194.519860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775194.519866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775194/real 1775775194] req@ffff9829764b5500 x1862032664701184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775194.519873:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664701184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775194.519878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775194.519880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775194.519881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775195.031171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775195.287227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775196.055163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775196.311220:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775197.079155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775197.335234:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775198.103161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775198.359245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775199.127142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775199.383227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775199.639186:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639196:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775199.639201:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775199.639211:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00080000:3.0:1775775199.639224:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775199.639230:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775199.639266:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664701312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775199.639291:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775199.639297:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775199.639302:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00080000:3.0:1775775199.639305:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775199.639311:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775199.639321:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775199.639336:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664701312:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775199.639337:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664701440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775199.639360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775199.639366:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775199.639371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00080000:3.0:1775775199.639375:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775199.639379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775199.639385:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775199.639395:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775199.639399:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664701568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775199.639411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775775199.639425:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9d500 x1862032664701312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775199.639426:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775199.639437:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00080000:3.0:1775775199.639443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775199.639446:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664701312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775199.639451:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775199.639460:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775199.639463:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775199.639465:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639471:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664701312:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775199.639482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664701696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775199.639492:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701440:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775199.639497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639502:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775199.639504:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775199.639511:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16967 00000800:00000100:0.0:1775775199.639517:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775199.639527:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9fb80 x1862032664701440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775199.639540:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664701440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775199.639546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775199.639548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639551:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775199.639559:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701568:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775199.639560:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775199.639569:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775199.639576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775775199.639576:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775199.639586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9ed80 x1862032664701568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775199.639594:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664701824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775199.639597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664701568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775199.639625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775199.639628:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639633:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775199.639634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775199.639638:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775199.639640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701568:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775199.639648:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00080000:3.0:1775775199.639653:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775199.639656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701696:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775199.639661:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775199.639668:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775199.639677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775199.639685:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664701952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775199.639688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9dc00 x1862032664701696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775199.639720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664701696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775199.639722:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775199.639733:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775199.639737:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775199.639744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775199.639746:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701696:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775199.639756:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16967 00000100:00100000:0.0:1775775199.639762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701824:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775199.639762:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775199.639771:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775199.639777:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775199.639791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775199.639795:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664702080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775199.639807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9f480 x1862032664701824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775199.639832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664701824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775199.639839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775199.639841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639844:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664701824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775199.639858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775199.639880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775199.639890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9f100 x1862032664701952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775199.639904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664701952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775199.639913:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775199.639916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664701952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775199.639926:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775199.639965:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775199.639974:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775199/real 1775775199] req@ffff982980f9c380 x1862032664702080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775199.639982:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664702080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775199.639987:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775199.639989:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775199.639991:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775200.151168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775200.407202:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775201.175162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775201.431205:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775202.199157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775202.455258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775203.223177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775203.479235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775204.247147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775204.503214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775204.759167:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775204.759176:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775204.759181:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759192:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759204:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759209:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775204.759248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664702208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775204.759281:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775204.759286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775204.759288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759294:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775204.759312:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775204.759328:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664702336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775204.759342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664702208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775204.759354:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775204.759358:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775204.759360:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759366:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759371:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759375:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775204.759382:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775204.759396:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664702464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775204.759407:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775204.759411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775204.759416:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775204.759419:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759438:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759449:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9f800 x1862032664702208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775204.759453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775204.759474:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664702592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775204.759479:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664702208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759500:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759504:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759509:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664702208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775204.759509:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775204.759522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775204.759525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759530:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759535:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759540:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759544:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702336:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775204.759557:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775204.759568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775204.759576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664702720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775204.759582:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9c700 x1862032664702336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775204.759604:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775204.759605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664702336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702336:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775204.759623:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775204.759626:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702464:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775204.759628:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775204.759634:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00080000:3.0:1775775204.759640:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759646:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775204.759653:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775204.759656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759666:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9e680 x1862032664702464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775204.759673:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664702848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775204.759683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664702464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775204.759688:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775204.759693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775204.759694:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775204.759698:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775204.759699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702464:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775204.759709:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16972 00000100:00100000:0.0:1775775204.759714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702592:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775204.759717:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775204.759724:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775204.759730:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775204.759733:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9ea00 x1862032664702592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775204.759750:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664702976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775204.759768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664702592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775204.759803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775204.759815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759824:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9c000 x1862032664702720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775204.759833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664702720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664702720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775204.759846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775204.759874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759881:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9d180 x1862032664702848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775204.759889:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664702848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759899:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775204.759902:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775204.759914:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775204.759919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775204/real 1775775204] req@ffff982980f9df80 x1862032664702976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775204.759926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664702976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775204.759931:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775204.759933:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775204.759935:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664702976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775205.271424:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775205.527214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775206.295150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775206.551233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775207.319193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775207.575292:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775208.343212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775208.599309:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775209.367164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775209.623235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775209.879134:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879145:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775209.879150:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879166:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00080000:3.0:1775775209.879179:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879185:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775209.879233:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664703104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775209.879258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879265:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775209.879267:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879271:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00080000:3.0:1775775209.879275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879279:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775209.879288:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775209.879303:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664703232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775209.879314:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879319:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775209.879321:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664703104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775209.879322:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879328:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00080000:3.0:1775775209.879336:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879339:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775209.879344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775209.879357:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664703360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775209.879368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879372:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775209.879374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879379:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00080000:3.0:1775775209.879383:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879387:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775209.879392:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775775209.879395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775209.879410:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664703488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775209.879421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775209.879424:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9d880 x1862032664703104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775209.879437:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775209.879441:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664703104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775209.879444:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879453:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00080000:3.0:1775775209.879460:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775209.879475:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775209.879478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879485:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775209.879490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664703616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775209.879494:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664703104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775209.879510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879514:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775209.879516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879520:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00100000:0.0:1775775209.879525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703232:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775209.879525:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775209.879533:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775209.879539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775209.879551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664703744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775209.879555:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9e300 x1862032664703232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775209.879568:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775209.879579:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775209.879583:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775209.879590:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16977 00000100:00100000:0.0:1775775209.879591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664703232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775209.879595:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775209.879602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775209.879603:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775209.879609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775775209.879611:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775209.879613:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775209.879621:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775209.879628:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664703872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775209.879638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9ce00 x1862032664703360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775209.879659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664703360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775209.879665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775209.879670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775209.879674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775209.879685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879691:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9df80 x1862032664703488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775209.879698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664703488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775209.879703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879704:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775209.879706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775209.879710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775209.879719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9d180 x1862032664703616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775209.879732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664703616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775209.879736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775209.879740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664703616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775209.879750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664703744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775209.879760:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879784:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9c000 x1862032664703744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775209.879792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664703744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775209.879798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775209.879802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664703744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775209.879805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664703872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775209.879817:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775209.879823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775209/real 1775775209] req@ffff982980f9ea00 x1862032664703872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775209.879831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664703872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775209.879836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775209.879837:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775209.879839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664703872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775210.391162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775210.647276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775211.415157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775211.671236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775212.439150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775212.695233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775213.463179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775213.719263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775214.487154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775214.743222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775214.999183:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999192:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775214.999196:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999205:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999215:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999221:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775214.999255:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664704000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775214.999279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999282:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775214.999283:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999286:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999289:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999291:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775214.999298:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775214.999306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664704128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775214.999314:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999316:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775214.999318:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999320:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999322:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775214.999327:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775214.999334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664704256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775214.999340:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664704000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775214.999343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999350:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775214.999351:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999355:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999367:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999373:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775214.999377:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775214.999386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664704384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775214.999392:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999395:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775214.999397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999399:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999402:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999404:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775214.999407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775214.999411:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775214.999418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664704512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775214.999426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775214.999430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775214.999432:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999438:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775214.999441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775214.999446:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe3b80 x1862032664704000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775214.999450:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664704640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775214.999470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775214.999473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775214.999475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775214.999476:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664704000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775214.999481:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16982 00000100:00080000:1.0:1775775214.999485:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775214.999492:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775214.999493:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775775214.999494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775214.999499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775775214.999504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664704768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775214.999505:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664704000:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775214.999517:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775214.999540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe1f80 x1862032664704128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999566:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664704128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999582:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775214.999604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775214.999623:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999633:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe0a80 x1862032664704256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664704256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999651:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999653:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775214.999659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775214.999672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999699:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe0e00 x1862032664704384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664704384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775214.999728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775214.999750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe1c00 x1862032664704512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664704512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664704512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775214.999796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664704640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775214.999817:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe1500 x1862032664704640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664704640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664704640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775214.999859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664704768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775214.999879:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775214.999889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775214/real 1775775214] req@ffff982981fe1180 x1862032664704768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775214.999900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664704768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775214.999908:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775214.999911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775214.999914:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664704768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775215.511203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775215.767277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775216.535136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775216.791215:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775217.559159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775217.815268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775218.583133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775218.839226:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775219.607174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775219.863250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775220.119191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119199:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775220.119203:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119212:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119229:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119234:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775220.119266:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664704896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119289:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119292:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775220.119293:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119296:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119299:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119301:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119309:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775220.119317:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664705024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119328:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775220.119329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119336:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119339:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775220.119349:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664704896:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775220.119351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664705152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119364:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119366:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775220.119367:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119370:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119372:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119375:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775220.119387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664705280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119394:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775220.119397:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119400:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119402:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119404:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775220.119413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664705408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775220.119422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119425:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119427:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775220.119430:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775220.119432:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119437:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775220.119443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664705536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775220.119454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775220.119456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775220.119458:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775220.119460:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16987 00000100:00080000:3.0:1775775220.119463:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775220.119465:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775220.119469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775220.119476:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9e680 x1862032664704896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775220.119477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664705664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775220.119520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664704896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119530:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119537:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664704896:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775220.119543:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775220.119566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9c700 x1862032664705024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119609:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664705024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119618:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119622:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119626:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775220.119660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775220.119677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9f800 x1862032664705152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664705152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119711:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775220.119720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775220.119743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9c380 x1862032664705280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664705280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119772:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119776:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775220.119780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775220.119795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9f100 x1862032664705408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664705408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775220.119826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664705536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775220.119835:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9f480 x1862032664705536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664705536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664705536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775220.119859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664705664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775220.119868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775220.119873:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775220/real 1775775220] req@ffff982980f9dc00 x1862032664705664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775220.119880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664705664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775220.119885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775220.119886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775220.119889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664705664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775220.631120:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775220.887251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775221.655215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775221.911269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775222.679186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775222.935209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775223.703211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775223.959261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775224.727167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775224.983287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775225.239130:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239141:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775225.239145:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239157:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239170:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239177:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775225.239234:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664705792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775225.239257:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239260:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775225.239261:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239266:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239274:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239279:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239289:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775225.239306:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664705920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775225.239307:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664705792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775225.239326:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239330:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775225.239333:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239337:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239347:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239353:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775225.239365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664706048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775225.239377:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775225.239383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239388:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239393:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775225.239398:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775225.239403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239410:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775225.239438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664706176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775225.239449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775225.239457:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9ed80 x1862032664705792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775225.239459:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239469:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239475:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239479:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775225.239491:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664705792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775225.239516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664706304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775225.239518:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239529:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775225.239531:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775225.239537:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664705792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775225.239538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239546:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239549:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239552:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239556:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775225.239564:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664706432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775225.239573:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705920:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775225.239573:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775225.239581:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775225.239584:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775225.239587:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16992 00000100:00080000:3.0:1775775225.239591:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775225.239593:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775225.239597:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775225.239604:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664706560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775225.239622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239639:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9fb80 x1862032664705920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664705920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.239662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.239668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664705920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775225.239675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775225.239698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9d500 x1862032664706048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664706048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.239733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239737:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.239740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775225.239747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775225.239769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9dc00 x1862032664706176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664706176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.239801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.239807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775225.239813:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775225.239832:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239843:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9f480 x1862032664706304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664706304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.239863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.239869:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775225.239889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664706432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775225.239910:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9f100 x1862032664706432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239934:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664706432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.239942:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.239945:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.239949:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664706432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775225.239954:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664706560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775225.239974:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775225.239985:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775225/real 1775775225] req@ffff982980f9c380 x1862032664706560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775225.239996:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664706560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775225.240004:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775225.240007:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775225.240010:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664706560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775225.751205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775226.007311:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775226.775218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775227.031258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775227.799183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775228.055255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775228.823231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775229.079260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775229.847201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775230.103240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775230.359155:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359164:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775230.359169:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359181:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359205:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359213:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775230.359249:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664706688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359278:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359282:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775230.359284:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359289:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359293:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359299:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775230.359310:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664706688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775230.359325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664706816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359348:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775230.359349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359353:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359358:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359362:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359368:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775230.359380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664706944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359393:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775230.359395:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359399:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359421:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775230.359436:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775230.359447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664707072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775230.359468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00100000:3.0:1775775230.359477:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706944:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775230.359481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359489:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359495:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775230.359508:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664707200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359519:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775230.359524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707072:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775775230.359525:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775230.359529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359552:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359563:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775230.359563:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775230.359582:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664707328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775230.359596:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775230.359600:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775230.359603:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664707328:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775230.359604:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775230.359614:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 16997 00000100:00080000:0.0:1775775230.359619:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775230.359625:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775230.359631:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775230.359644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664707456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775230.359692:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664707456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775230.871183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775231.127300:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775231.895149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775232.151236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775232.919161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775233.175267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775233.431318:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775233.431390:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775233.431411:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431458:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431467:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431474:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431482:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431493:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775233.431503:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775233.431526:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2df80 x1862032664707456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431555:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664707456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431568:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431574:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431579:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664707456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775233.431590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2f100 x1862032664707328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664707328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431617:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664707328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775233.431627:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2ca80 x1862032664707072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431640:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664707072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775233.431662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2fb80 x1862032664707200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664707200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775233.431697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2f480 x1862032664706816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664706816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431722:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431730:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775233.431737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2d500 x1862032664706944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431748:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664706944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431756:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431759:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664706944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775233.431769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775230/real 1775775233] req@ffff982974b2d180 x1862032664706688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775233.431781:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664706688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775233.431789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775233.431793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775233.431797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664706688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775233.943145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775234.199225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775234.967222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775235.223248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775235.479225:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775235.479237:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479245:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479256:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479261:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775235.479292:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664707584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775235.479332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775235.479339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775235.479379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664707712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775235.479391:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775235.479398:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664707584:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775235.479401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479412:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479419:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479423:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775235.479445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664707840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775235.479456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479460:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775235.479461:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479466:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479470:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479474:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479481:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775235.479490:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775235.479495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664707968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775235.479507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479511:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775235.479512:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479516:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479521:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479525:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479531:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775235.479533:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe1180 x1862032664707584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775235.479545:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664708096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775235.479555:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479560:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775235.479562:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479567:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00100000:3.0:1775775235.479568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664707584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775235.479572:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775235.479580:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479588:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775235.479588:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.479597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775775235.479605:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664708224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775235.479605:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664707584:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775235.479623:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775235.479626:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775235.479628:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775235.479634:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17002 00000100:00080000:1.0:1775775235.479638:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775235.479643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707712:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775235.479645:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775235.479655:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775235.479670:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664708352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775235.479684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.479703:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe1500 x1862032664707712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.479720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664707712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.479729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.479733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.479737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775235.479743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775235.479766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.479778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe1c00 x1862032664707840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.479792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664707840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.479800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.479804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.479808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775235.479814:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775235.479838:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.479849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe0e00 x1862032664707968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.479861:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664707968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.479870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.479873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.479877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664707968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775235.479884:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775235.479904:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.479915:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe0a80 x1862032664708096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.479926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664708096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.479934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.479938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.479941:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775235.479947:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664708224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775235.480105:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.480128:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe1f80 x1862032664708224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.480140:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664708224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.480149:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.480152:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.480156:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664708224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775235.480178:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664708352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775235.480200:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775235.480211:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775235/real 1775775235] req@ffff982981fe3b80 x1862032664708352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775235.480223:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664708352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775235.480231:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775235.480234:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775235.480237:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664708352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775235.991148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775236.247216:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775237.015180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775237.271249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775238.039131:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775238.295290:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775239.063142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775239.319297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775240.087184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775240.343204:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775240.599172:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599180:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775240.599183:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775240.599192:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599202:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599208:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775240.599267:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664708480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599294:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599297:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775240.599298:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775240.599301:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599304:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599307:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775240.599315:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775240.599324:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664708608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599335:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775240.599337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775240.599341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664708480:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775240.599342:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599348:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775240.599354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775240.599364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664708736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775240.599374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775240.599377:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599379:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599381:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775240.599384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775240.599391:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664708864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775240.599401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775240.599403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775240.599410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775240.599415:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775240.599419:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664708992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599429:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775240.599430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775240.599438:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599441:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599445:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775240.599449:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775240.599456:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9f800 x1862032664708480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775240.599457:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664709120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775240.599491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775240.599494:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775240.599495:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775240.599501:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664708480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775240.599501:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17008 00000100:00080000:3.0:1775775240.599506:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775240.599512:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775240.599519:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775240.599519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775240.599527:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599529:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664708480:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775240.599538:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664709248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775240.599552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775240.599573:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9c700 x1862032664708608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599593:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664708608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775240.599607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775240.599618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599624:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9e680 x1862032664708736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599631:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664708736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775240.599644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775240.599654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599660:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9ea00 x1862032664708864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664708864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599674:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775240.599679:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775240.599689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9c000 x1862032664708992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664708992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664708992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775240.599713:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664709120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775240.599722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9d180 x1862032664709120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599734:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664709120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599739:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664709120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775240.599755:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664709248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775240.599765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775240.599771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775240/real 1775775240] req@ffff982980f9df80 x1862032664709248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775240.599783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664709248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775240.599788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775240.599789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775240.599791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664709248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775241.111174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775241.367252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775242.135124:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775242.391282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775243.159178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775243.415291:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775244.183186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775244.439240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775245.207194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775245.463225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775245.719196:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719204:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775245.719209:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719216:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719235:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775245.719268:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664709376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775245.719287:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775245.719291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719294:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719297:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719299:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775245.719308:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775245.719316:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664709504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775245.719323:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719326:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775245.719327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775245.719333:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664709376:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775245.719333:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719336:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719343:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775245.719347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775245.719361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664709632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775245.719371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775245.719377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719385:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719389:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775245.719395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775245.719409:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664709760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775245.719410:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775245.719421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775245.719428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719433:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719437:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775245.719446:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775245.719450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b5f80 x1862032664709376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775245.719460:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664709888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775245.719473:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775245.719478:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664709376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775245.719479:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775245.719482:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719492:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719498:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775245.719502:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775245.719503:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775245.719510:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775775245.719511:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775245.719515:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664709376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775245.719526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664710016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775245.719544:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709504:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775245.719548:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775245.719552:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775245.719554:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775245.719560:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17013 00000100:00080000:2.0:1775775245.719566:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775245.719570:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775245.719575:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775245.719576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775245.719591:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664710144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775245.719594:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b4700 x1862032664709504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664709504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719639:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719643:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719646:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775245.719652:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775245.719671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775245.719680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b5c00 x1862032664709632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664709632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775245.719712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775245.719728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775245.719738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b5180 x1862032664709760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719749:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664709760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775245.719775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775245.719792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775245.719800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b4a80 x1862032664709888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664709888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664709888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775245.719842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775245.719859:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775245.719869:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b6d80 x1862032664710016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664710016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719890:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719897:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775245.719902:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775245.719919:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775245.719928:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775245/real 1775775245] req@ffff9829764b5f80 x1862032664710144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775245.719940:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664710144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775245.719948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775245.719951:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775245.719954:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775246.231203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775246.487268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775247.255183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775247.511267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775248.279208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775248.535267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775249.303187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775249.559304:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775250.327197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775250.583264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775250.839206:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839214:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775250.839219:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839234:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839245:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775250.839251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775250.839284:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664710272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775250.839308:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839311:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775250.839312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839315:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839317:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775250.839320:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775250.839329:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775250.839338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664710400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775250.839346:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839348:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775250.839349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839352:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839354:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775250.839358:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664710272:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775250.839359:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775250.839365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775250.839373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664710528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775250.839379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775250.839383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839385:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839387:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775250.839390:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775250.839395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775250.839402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664710656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775250.839408:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775250.839413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839415:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839419:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775250.839423:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775250.839426:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775250.839430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775250.839438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664710784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775250.839445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839447:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775250.839448:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775250.839452:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839454:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775250.839458:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe0380 x1862032664710272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775250.839460:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775250.839467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775250.839478:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664710912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775250.839482:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664710272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775250.839492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775250.839498:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775250.839500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775250.839501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775250.839505:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17018 00000100:00080000:1.0:1775775250.839508:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775250.839509:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775250.839514:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839516:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664710272:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775250.839519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775250.839530:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775250.839534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664711040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775250.839553:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe0000 x1862032664710400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664710400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839582:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839584:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775250.839605:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775250.839618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe1880 x1862032664710528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664710528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839639:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839643:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775250.839648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775250.839662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe2300 x1862032664710656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664710656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839686:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839691:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775250.839698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775250.839718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe2680 x1862032664710784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839742:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664710784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664710784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775250.839764:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775250.839779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe2a00 x1862032664710912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664710912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839803:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664710912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775250.839812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775250.839831:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775250.839842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775250/real 1775775250] req@ffff982981fe3480 x1862032664711040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775250.839854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664711040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775250.839863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775250.839866:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775250.839868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775251.351209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775251.607231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775252.375173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775252.631265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775253.399163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775253.655306:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775254.423206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775254.679440:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775255.447194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775255.703260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775255.959161:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959168:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775255.959172:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959180:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959190:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959196:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775255.959235:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664711168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959265:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959268:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775255.959269:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959273:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959276:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959279:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959287:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775255.959296:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664711296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959303:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959306:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775255.959307:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959310:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959313:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959316:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959322:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775255.959332:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664711424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775255.959347:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959351:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959355:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959359:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959366:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775255.959370:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664711168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775255.959380:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664711552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959391:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775255.959392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959395:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959397:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959400:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775255.959411:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664711680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959420:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775255.959422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959424:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959426:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959428:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775255.959437:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664711808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775255.959442:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775255.959445:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775255.959446:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775255.959449:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17023 00000100:00080000:1.0:1775775255.959470:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775255.959474:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775255.959477:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775255.959484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664711936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775255.959494:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.959557:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe3480 x1862032664711168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.959581:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664711168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.959608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.959614:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.959619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664711168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775255.959629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775255.959657:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.959687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe2a00 x1862032664711296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.959701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664711296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.959709:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.959713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.959716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775255.959723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775255.959755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.959766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe2680 x1862032664711424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.959778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664711424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.959786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.959790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.959795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775255.959826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775255.959850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.959861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe2300 x1862032664711552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.959873:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664711552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.959880:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.959884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.959888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775255.959894:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775255.959912:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.959922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe1880 x1862032664711680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.959934:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664711680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.959943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.959946:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.959949:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664711680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775255.959956:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775255.959990:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.960002:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe0000 x1862032664711808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.960014:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664711808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.960022:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.960025:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.960028:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775255.960034:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775255.960054:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775255.960107:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775255/real 1775775255] req@ffff982981fe0380 x1862032664711936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775255.960136:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664711936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775255.960144:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775255.960147:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775255.960150:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664711936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775256.471182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775256.727240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775257.495143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775257.751263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775258.519164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775258.775290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775259.543145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775259.799257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775260.567205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775260.823295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775261.079211:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079219:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775261.079223:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079233:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079246:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079253:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775261.079290:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664712064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775261.079320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775261.079324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775261.079353:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664712192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775261.079360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079363:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775261.079364:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079367:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079369:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079373:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775261.079378:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664712064:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775261.079389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664712320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775261.079401:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079404:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775261.079405:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079408:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079411:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079413:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079417:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775261.079427:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664712448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775261.079434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079437:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775261.079438:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079440:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775261.079448:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775261.079450:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079457:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775261.079470:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664712576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775261.079481:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079485:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775261.079489:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775261.079489:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2c380 x1862032664712064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775261.079499:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079507:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079513:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079519:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775261.079520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664712064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079534:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775261.079537:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664712704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775261.079542:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079551:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664712064:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775261.079557:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775261.079562:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775261.079563:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712192:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775261.079565:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775261.079574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17028 00000100:00080000:0.0:1775775261.079577:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775261.079580:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775261.079585:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775261.079598:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664712832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775261.079600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2e680 x1862032664712192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079631:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664712192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775261.079671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775261.079690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079700:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2e300 x1862032664712320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079712:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664712320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775261.079733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775261.079752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2d180 x1862032664712448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079773:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664712448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775261.079794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775261.079810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2d500 x1862032664712576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664712576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079842:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664712576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775261.079851:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664712704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775261.079867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079896:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2f480 x1862032664712704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664712704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079923:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664712704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775261.079929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664712832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775261.079947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775261.079956:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775261/real 1775775261] req@ffff982974b2fb80 x1862032664712832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775261.079975:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664712832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775261.079982:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775261.079985:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775261.079988:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664712832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775261.591170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775261.847232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775262.615176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775262.871251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775263.639184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775263.895221:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775264.663187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775264.919239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775265.687175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775265.943248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775266.199175:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199183:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775266.199189:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199197:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199207:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199213:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775266.199245:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664712960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199265:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199268:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775266.199269:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199272:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199278:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199286:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775266.199294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664713088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199301:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199305:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775266.199306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775266.199307:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664712960:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775266.199310:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199322:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775266.199330:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664713216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775266.199340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199342:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199345:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199347:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199352:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775266.199359:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664713344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775266.199369:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199371:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199373:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199375:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775266.199381:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775266.199394:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664713472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199401:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775266.199405:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199407:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199410:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199416:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775266.199423:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2ca80 x1862032664712960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775266.199424:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664713600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775266.199445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775266.199451:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775266.199453:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775266.199456:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17033 00000100:00080000:0.0:1775775266.199459:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775266.199461:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775266.199464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775266.199470:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664712960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775266.199471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664713728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775266.199498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199504:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199510:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664712960:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775266.199547:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775266.199577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2f100 x1862032664713088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664713088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199618:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199622:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775266.199628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775266.199651:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2df80 x1862032664713216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199676:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664713216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199685:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199691:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775266.199698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775266.199719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2fb80 x1862032664713344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664713344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199759:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775266.199772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775266.199812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2f480 x1862032664713472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664713472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775266.199859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664713600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775266.199877:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199885:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2d500 x1862032664713600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199897:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664713600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199912:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664713600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775266.199924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664713728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775266.199938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775266.199946:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775266/real 1775775266] req@ffff982974b2d180 x1862032664713728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775266.199957:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664713728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775266.199965:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775266.199969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775266.199972:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664713728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775266.711193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775266.967256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775267.735220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775267.991262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775268.759173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775269.015222:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775269.783196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775270.039223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775270.807202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775271.063204:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775271.319163:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319171:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775271.319175:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319182:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319192:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775271.319197:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775271.319224:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664713856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775271.319244:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775271.319247:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319250:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319252:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775271.319254:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775271.319262:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775271.319270:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664713984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775271.319276:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319278:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775271.319279:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319281:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319283:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775271.319287:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319287:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664713856:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775271.319289:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775271.319299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664714112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775271.319304:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319306:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775271.319307:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319309:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319311:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775271.319313:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775271.319316:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775271.319324:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664714240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775271.319328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319331:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775271.319333:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319335:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319337:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775271.319341:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775271.319342:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775271.319347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775271.319353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664714368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775271.319360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775271.319365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775271.319369:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775271.319377:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2e300 x1862032664713856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775271.319380:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775271.319387:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775271.319402:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664714496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775271.319410:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664713856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775271.319413:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775271.319422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775271.319425:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775271.319429:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775271.319431:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17038 00000100:00080000:0.0:1775775271.319436:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775271.319438:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775271.319442:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775271.319447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775271.319455:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664713856:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775271.319465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664714624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775271.319465:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775271.319489:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319505:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2e680 x1862032664713984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319512:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664713984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319517:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319520:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664713984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775271.319536:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775271.319546:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2c380 x1862032664714112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664714112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319569:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319571:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775271.319574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775271.319583:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2d180 x1862032664714240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319594:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664714240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319598:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775271.319604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775271.319613:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2d500 x1862032664714368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664714368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319629:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319631:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775271.319634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664714496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775271.319642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2f480 x1862032664714496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664714496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319656:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319658:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664714496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775271.319662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664714624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775271.319670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775271.319674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775271/real 1775775271] req@ffff982974b2fb80 x1862032664714624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775271.319680:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664714624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775271.319684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775271.319685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775271.319687:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664714624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775271.831198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775272.087276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775272.855148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775273.111231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775273.879149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775274.135233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775274.903135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775275.159191:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775275.927156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775276.183231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775276.439171:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439180:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775276.439185:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439197:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439210:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439219:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775276.439268:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664714752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439291:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439294:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775276.439296:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439299:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439302:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439306:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439314:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775276.439323:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664714880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775276.439336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00100000:3.0:1775775276.439342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664714752:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775276.439343:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775276.439362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664715008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775276.439372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439374:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439377:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439380:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775276.439390:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664715136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775276.439399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439402:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439406:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775276.439415:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775276.439421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664715264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775276.439433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775276.439435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439437:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439441:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775276.439450:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2df80 x1862032664714752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775276.439453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664715392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775276.439475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775276.439478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775276.439479:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775276.439483:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664714752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775276.439486:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17043 00000100:00080000:0.0:1775775276.439492:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775276.439498:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775276.439502:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775276.439503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775276.439509:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664715520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775276.439511:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439520:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664714752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775276.439532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775276.439562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439576:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2f100 x1862032664714880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664714880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439614:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664714880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775276.439659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775276.439683:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2ca80 x1862032664715008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664715008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775276.439732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775276.439754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2fb80 x1862032664715136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664715136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439787:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775276.439800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775276.439821:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2f480 x1862032664715264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664715264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439855:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439863:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775276.439869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664715392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775276.439890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439901:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2d500 x1862032664715392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439913:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664715392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439921:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664715392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775276.439935:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664715520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775276.439955:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775276.439967:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775276/real 1775775276] req@ffff982974b2d180 x1862032664715520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775276.439979:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664715520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775276.439988:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775276.439991:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775276.439995:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664715520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775276.951138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775277.207240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775277.975170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775278.231248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775278.999144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775279.255264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775280.023188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775280.279262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775281.047158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775281.303232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775281.559202:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559210:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775281.559215:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559224:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559234:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559240:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775281.559275:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664715648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775281.559310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775281.559317:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559322:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559326:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775281.559341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775281.559355:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664715776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775281.559373:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664715648:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775281.559382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775281.559387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559390:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559397:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775281.559401:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775281.559412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664715904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775281.559418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775281.559422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559427:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559429:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775281.559433:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775281.559437:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775281.559442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664716032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775281.559450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559452:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775281.559454:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559457:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559459:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559462:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775281.559465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775281.559467:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe3b80 x1862032664715648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775281.559474:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664716160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775281.559484:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559487:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775281.559490:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775281.559490:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664715648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775281.559498:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559503:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775281.559509:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775281.559509:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559529:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664715648:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775281.559544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775281.559554:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775281.559560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664716288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775281.559574:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775281.559578:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775775281.559580:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775281.559582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775281.559587:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17049 00000100:00080000:1.0:1775775281.559592:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775281.559595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe1f80 x1862032664715776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775281.559599:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775281.559607:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775281.559621:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664716416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775281.559623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664715776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559656:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775281.559664:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775281.559689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775281.559700:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe0a80 x1862032664715904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775281.559710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664715904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664715904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775281.559725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775281.559740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775281.559746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe0e00 x1862032664716032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775281.559754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664716032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775281.559771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775281.559780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775281.559785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe1c00 x1862032664716160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775281.559792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664716160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559797:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775281.559814:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664716288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775281.559824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775281.559830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe1500 x1862032664716288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775281.559837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664716288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664716288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775281.559850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664716416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775281.559860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775281.559865:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775281/real 1775775281] req@ffff982981fe1180 x1862032664716416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775281.559872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664716416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775281.559877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775281.559879:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775281.559880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664716416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775282.071139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775282.327252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775283.095185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775283.351203:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775284.119172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775284.375233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775285.143187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775285.399173:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775286.167163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775286.423252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775286.679196:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679205:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775286.679210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679218:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679233:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775286.679265:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664716544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775286.679295:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679298:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775286.679299:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679302:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679305:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679308:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775286.679321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775286.679331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664716672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775286.679337:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775286.679341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775286.679352:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775286.679361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664716800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775286.679362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664716544:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775286.679372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775286.679376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679384:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679387:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775286.679391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775286.679400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664716928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775286.679406:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775286.679410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679413:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679415:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679417:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775286.679422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775286.679446:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664717056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775286.679455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679459:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775286.679463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775286.679464:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775286.679470:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00080000:3.0:1775775286.679475:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679480:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775286.679486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775286.679499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664717184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775286.679507:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9ce00 x1862032664716544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775286.679523:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775286.679526:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775286.679528:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775286.679531:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17054 00000100:00100000:0.0:1775775286.679534:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664716544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775286.679536:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775286.679544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775286.679548:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775286.679549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775286.679554:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679559:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664716544:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775286.679562:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664717312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775286.679569:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775286.679592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679601:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9e300 x1862032664716672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679609:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664716672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775286.679638:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775286.679652:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679658:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9d880 x1862032664716800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679665:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664716800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679672:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775286.679677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775286.679688:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9df80 x1862032664716928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664716928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664716928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775286.679712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775286.679722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9d180 x1862032664717056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679739:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664717056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775286.679750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664717184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775286.679759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9c000 x1862032664717184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664717184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664717184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775286.679783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664717312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775286.679792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775286.679797:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775286/real 1775775286] req@ffff982980f9ea00 x1862032664717312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775286.679804:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664717312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775286.679813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775286.679815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775286.679817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664717312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775287.191212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775287.447197:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775288.215162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775288.471284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775289.239189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775289.495246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775290.263136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775290.519202:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775291.287165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775291.543250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775291.799144:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775291.799154:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775291.799158:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799170:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799183:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775291.799190:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775291.799235:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664717440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775291.799264:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775291.799268:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775291.799270:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799275:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799280:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775291.799285:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775291.799295:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775291.799306:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664717440:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775291.799311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664717568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775291.799332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775291.799349:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775291.799351:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799355:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799359:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775291.799363:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775291.799369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775291.799381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664717696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775291.799391:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775291.799396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775291.799400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775291.799402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799425:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775291.799430:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2c380 x1862032664717440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775291.799432:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775291.799441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775291.799459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664717824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775291.799463:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664717440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775291.799486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775291.799487:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775291.799494:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775291.799494:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775291.799499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775291.799501:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664717440:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775291.799506:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799512:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775291.799517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775291.799521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775291.799533:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775291.799534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664717952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775291.799550:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775291.799554:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775291.799557:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799561:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:0.0:1775775291.799565:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775291.799571:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775291.799573:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775291.799578:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775291.799590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664718080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775291.799595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2e680 x1862032664717568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775291.799615:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775291.799620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664717568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775291.799621:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775291.799624:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775291.799633:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17059 00000100:00080000:3.0:1775775291.799635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775291.799638:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775291.799640:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775291.799647:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717568:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775291.799656:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775291.799658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775291.799674:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664718208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775291.799696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2e300 x1862032664717696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775291.799717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664717696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775291.799723:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775291.799725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775291.799727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775291.799731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775291.799744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2d180 x1862032664717824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775291.799757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664717824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775291.799763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775291.799765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775291.799767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775291.799779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775291.799789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799795:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2d500 x1862032664717952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775291.799802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664717952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775291.799807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775291.799809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775291.799811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664717952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775291.799814:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775291.799823:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2f480 x1862032664718080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775291.799835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664718080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775291.799840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775291.799841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775291.799844:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775291.799847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775291.799856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775291.799861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775291/real 1775775291] req@ffff982974b2fb80 x1862032664718208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775291.799868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664718208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775291.799873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775291.799875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775291.799876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775292.311147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775292.567232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775293.335173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775293.591249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775294.359164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775294.615218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775295.383170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775295.639262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775296.407147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775296.663234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775296.919162:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919174:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775296.919179:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919194:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919206:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919212:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775296.919269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664718336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775296.919300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775296.919306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919311:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919317:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919322:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775296.919333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775296.919341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664718336:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775296.919359:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664718464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775296.919372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775296.919378:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919396:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775296.919402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775296.919431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664718592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775296.919443:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718464:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775296.919446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775296.919455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919461:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919465:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919470:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775296.919477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775296.919490:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775296.919495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664718720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775296.919508:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775296.919514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919519:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00100000:0.0:1775775296.919524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718720:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775296.919525:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919534:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775296.919540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775296.919563:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664718848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775296.919583:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919587:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775296.919590:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775296.919594:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919599:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919603:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775296.919612:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718848:192.168.201.118@tcp:8:kworker.0 00000008:00100000:3.0:1775775296.919617:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775296.919635:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664718976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775296.919646:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775296.919649:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775296.919650:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775296.919653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718976:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775296.919657:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17064 00000100:00080000:3.0:1775775296.919664:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775296.919668:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775296.919673:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775296.919683:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664719104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775296.919723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664719104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775297.431177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775297.687280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775298.455143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775298.711210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775299.479168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775299.735259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775775299.991352:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775775299.991397:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775775299.991413:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991453:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991461:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991469:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991478:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991489:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775299.991500:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775299.991566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9dc00 x1862032664719104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664719104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664719104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775299.991646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9f480 x1862032664718976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664718976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664718976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775299.991686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9f100 x1862032664718848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664718848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775299.991728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9c380 x1862032664718720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991739:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664718720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775299.991761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9f800 x1862032664718592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664718592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775299.991798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9c700 x1862032664718464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664718464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664718464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775299.991836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775296/real 1775775299] req@ffff982980f9e680 x1862032664718336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775299.991850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664718336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775299.991858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775299.991863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775299.991868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664718336:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775300.503161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775300.759232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775301.527185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775301.783299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775302.039152:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039185:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775302.039190:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039201:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039215:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775302.039222:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775302.039265:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664719232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775302.039296:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039299:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775302.039301:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039304:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039307:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775302.039310:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775302.039319:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775302.039328:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664719360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775302.039337:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039339:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775302.039341:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039345:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039348:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775302.039354:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039357:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664719232:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775302.039361:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775302.039375:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664719488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775302.039386:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775302.039392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039397:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039401:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775302.039405:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775302.039413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775302.039429:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664719616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775302.039439:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039443:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775302.039445:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039449:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039454:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775302.039455:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775302.039461:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775302.039467:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775302.039481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664719744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775302.039494:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775302.039500:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775302.039505:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039510:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775302.039516:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039517:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2ca80 x1862032664719232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775302.039524:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775302.039546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664719872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775302.039552:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664719232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775302.039569:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775302.039575:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775302.039577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775302.039578:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775302.039589:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775302.039590:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17069 00000100:00080000:0.0:1775775302.039598:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775302.039600:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664719232:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775302.039607:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719360:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775302.039615:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775302.039635:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664720000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775302.039651:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2f100 x1862032664719360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.039683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664719360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.039693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.039698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.039702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775302.039739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775302.039764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2df80 x1862032664719488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.039787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664719488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.039804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.039808:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.039812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775302.039819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775302.039841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.039852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2fb80 x1862032664719616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.039863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664719616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.039871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.039875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.039878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775302.039883:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775302.039903:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.040867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2f480 x1862032664719744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.040895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664719744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.040906:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.040912:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.040918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664719744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775302.040931:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664719872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775302.040982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.041006:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2d500 x1862032664719872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.041021:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664719872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.041030:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.041035:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.041038:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664719872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775302.041046:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775302.041091:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775302.041104:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775302/real 1775775302] req@ffff982974b2d180 x1862032664720000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775302.041124:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664720000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775302.041133:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775302.041136:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775302.041141:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775302.551208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775302.807214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775303.575212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775303.831297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775304.599210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775304.855281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775305.623189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775305.879286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775306.647211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775306.903262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775307.159191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159205:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775307.159211:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159225:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159241:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775307.159250:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775307.159315:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664720128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775307.159353:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159358:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775307.159360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159366:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159371:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775307.159376:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775307.159388:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775307.159407:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664720256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775307.159419:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664720128:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775307.159421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159428:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775307.159430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159436:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159440:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775307.159445:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775307.159451:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775307.159467:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664720384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775307.159478:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159482:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775307.159485:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159489:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17074 00000800:00000100:3.0:1775775307.159494:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775307.159496:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775307.159502:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775307.159509:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775307.159525:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664720512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775307.159535:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2e300 x1862032664720128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775307.159537:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159547:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775307.159551:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159559:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159564:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775307.159564:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664720128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775307.159574:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775307.159580:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775307.159582:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775307.159587:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159591:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664720128:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775307.159599:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664720640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775307.159610:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775307.159615:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775775307.159615:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720256:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775307.159619:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159626:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159632:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775307.159636:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775307.159638:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775307.159646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775307.159650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2e680 x1862032664720256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775307.159663:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664720768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775307.159669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664720256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775307.159693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720256:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775307.159706:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775307.159711:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720384:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775307.159712:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775307.159716:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775307.159722:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17074 00000100:00080000:0.0:1775775307.159727:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775307.159730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775307.159736:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775307.159741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2c380 x1862032664720384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775307.159745:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775307.159755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664720384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775307.159764:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664720896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775307.159769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159780:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775307.159789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775307.159804:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775307.159812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2d180 x1862032664720512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775307.159820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664720512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775307.159830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775307.159837:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775307.159854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775307.159860:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2d500 x1862032664720640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775307.159867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664720640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775307.159873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664720640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775307.159888:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775307.159899:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775307.159905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2f480 x1862032664720768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775307.159912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664720768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775307.159918:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775307.159926:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775307.159936:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775307.159941:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775307/real 1775775307] req@ffff982974b2fb80 x1862032664720896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775307.159948:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664720896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775307.159953:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775307.159955:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775307.159957:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664720896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775307.671201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775307.927245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775308.695167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775308.951241:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775309.719161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775309.975225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775310.743192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775310.999236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775311.767158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775312.023211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775312.279190:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279197:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775312.279201:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279210:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279221:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279227:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775312.279260:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664721024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775312.279285:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279288:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775312.279290:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279293:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279295:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279298:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279306:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775312.279313:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664721152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775312.279321:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279324:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775312.279325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279335:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775312.279341:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664721280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775312.279348:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664721024:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775312.279351:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279356:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775312.279359:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279363:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279365:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279367:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279370:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775312.279379:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664721408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775312.279385:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279388:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775312.279389:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279392:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279394:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279396:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775312.279405:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664721536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775312.279412:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775312.279416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279418:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279420:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279422:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279425:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775312.279429:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775312.279436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664721664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775312.279443:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775312.279446:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775312.279447:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775312.279450:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17079 00000100:00080000:3.0:1775775312.279452:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775312.279455:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775312.279459:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775312.279467:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9d500 x1862032664721024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775312.279468:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664721792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775312.279508:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664721024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279516:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279520:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279523:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664721024:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775312.279529:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775312.279553:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9fb80 x1862032664721152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279582:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664721152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279591:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279595:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279599:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775312.279622:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775312.279638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9ed80 x1862032664721280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664721280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279663:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775312.279666:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775312.279677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9e680 x1862032664721408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664721408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279695:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775312.279703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775312.279712:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279717:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9c700 x1862032664721536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279724:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664721536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664721536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775312.279736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664721664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775312.279746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9f800 x1862032664721664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664721664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279762:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664721664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775312.279770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664721792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775312.279779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775312.279785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775312/real 1775775312] req@ffff982980f9c380 x1862032664721792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775312.279791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664721792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775312.279796:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775312.279798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775312.279801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664721792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775312.791168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775313.047205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775313.815186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775314.071252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775314.839155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775315.095231:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775315.863156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775316.119214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775316.887353:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775317.143316:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775317.399238:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399252:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775317.399258:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775317.399273:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:2.0:1775775317.399290:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775317.399299:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775317.399349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664721920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775317.399387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399391:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775317.399393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775317.399399:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:2.0:1775775317.399404:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775317.399409:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775317.399423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775317.399441:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664721920:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775317.399441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664722048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775317.399467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399471:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775317.399472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775317.399477:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:2.0:1775775317.399482:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775317.399486:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775317.399492:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775317.399505:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664722176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775317.399517:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399521:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775317.399523:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775317.399537:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17084 00000800:00000100:0.0:1775775317.399538:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775317.399546:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775317.399552:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775317.399559:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775317.399575:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664722304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775317.399581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b5500 x1862032664721920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775317.399602:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399607:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775317.399609:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775317.399611:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664721920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775317.399619:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:2.0:1775775317.399627:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775317.399631:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775317.399634:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775317.399639:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.399644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664721920:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775317.399644:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775317.399666:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664722432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775317.399678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775317.399680:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775317.399687:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775317.399689:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775317.399695:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:2.0:1775775317.399700:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775317.399704:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775317.399710:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775317.399723:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775317.399737:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664722560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775317.399741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b4000 x1862032664722048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775317.399767:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775317.399771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664722048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775317.399774:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775317.399779:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775317.399786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775317.399790:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17084 00000100:00080000:0.0:1775775317.399793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.399797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775317.399798:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775317.399806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722176:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775317.399807:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775317.399817:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775317.399833:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664722688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775317.399840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775317.399858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b6680 x1862032664722176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775317.399874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664722176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775317.399884:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775317.399902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.399905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775317.399913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775317.399938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775317.399950:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b4e00 x1862032664722304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775317.399971:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664722304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775317.399980:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775317.399983:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.399987:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775317.400007:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775317.400028:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775317.400037:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b7b80 x1862032664722432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775317.400046:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664722432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775317.400052:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775317.400054:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.400056:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775317.400060:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664722560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775317.400123:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775317.400136:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b7100 x1862032664722560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775317.400149:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664722560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775317.400165:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775317.400168:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.400172:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664722560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775317.400177:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664722688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775317.400198:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775317.400211:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775317/real 1775775317] req@ffff9829764b6a00 x1862032664722688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775317.400236:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664722688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775317.400245:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775317.400248:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775317.400251:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664722688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775317.911177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775318.167301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775318.935209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775319.191342:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775319.959236:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775320.215224:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775320.983174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775321.239283:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775322.007198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775322.263209:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775322.519208:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519219:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775322.519223:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519235:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519250:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519258:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775322.519308:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664722816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775322.519335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775322.519340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519348:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775322.519374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664722944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775322.519382:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664722816:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775322.519394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775322.519398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519402:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519406:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519408:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775322.519421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664723072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775322.519428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775322.519432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519435:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519438:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519440:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775322.519451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664723200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775322.519457:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519460:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775322.519462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519465:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519469:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775322.519470:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775322.519474:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519478:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775322.519486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664723328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775322.519494:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775322.519498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519501:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519504:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519507:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519511:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775322.519515:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9f100 x1862032664722816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775322.519519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664723456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775322.519525:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775322.519528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775322.519529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775322.519536:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17090 00000100:00080000:3.0:1775775322.519541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775322.519544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775322.519548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775322.519553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664722816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775322.519557:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664723584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775322.519576:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664722816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775322.519611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775322.519629:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9f480 x1862032664722944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519647:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664722944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519652:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519657:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664722944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775322.519660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775322.519671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519681:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9dc00 x1862032664723072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664723072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775322.519715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775322.519739:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9ea00 x1862032664723200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664723200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775322.519774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775322.519794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519837:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9c000 x1862032664723328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664723328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775322.519866:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664723456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775322.519882:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9d180 x1862032664723456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519896:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664723456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519908:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519911:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664723456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775322.519917:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664723584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775322.519939:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775322.519950:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775322/real 1775775322] req@ffff982980f9df80 x1862032664723584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775322.519963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664723584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775322.519970:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775322.519973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775322.519975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664723584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775323.031218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775323.287313:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775324.055194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775324.311195:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775325.079190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775325.335222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775326.103174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775326.359246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775327.127177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775327.383222:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775327.639149:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639159:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775327.639163:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639172:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639185:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639191:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775327.639223:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664723712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775327.639244:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775327.639248:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639250:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639253:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639255:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639263:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775327.639271:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664723840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775327.639281:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639283:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775327.639285:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639287:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639290:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639294:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775327.639303:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664723712:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775327.639311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664723968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775327.639321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639325:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775327.639326:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639330:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639334:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639338:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775327.639363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664724096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775327.639370:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775327.639377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775327.639382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639387:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639396:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639401:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775327.639406:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9d880 x1862032664723712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775327.639414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664724224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775327.639425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775327.639428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775327.639432:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664723712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775327.639434:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639438:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775327.639443:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639450:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775327.639451:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639458:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639464:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664723712:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775327.639464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664724352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775327.639476:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723840:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775327.639480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775327.639484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775327.639485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775327.639497:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17095 00000100:00080000:3.0:1775775327.639503:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775327.639507:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775327.639510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775327.639519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775327.639520:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9e300 x1862032664723840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639533:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664723840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775327.639533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664724480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775327.639561:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639567:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639573:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775327.639593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775327.639605:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775327.639610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9ce00 x1862032664723968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664723968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775327.639621:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664723968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775327.639628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775327.639644:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775327.639654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9df80 x1862032664724096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639665:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664724096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775327.639672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775327.639683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775327.639700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775327.639708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9d180 x1862032664724224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664724224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775327.639734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639737:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775327.639745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664724352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775327.639763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775327.639772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9c000 x1862032664724352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664724352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775327.639794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639798:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664724352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775327.639803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664724480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775327.639819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775327.639828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775327/real 1775775327] req@ffff982980f9ea00 x1862032664724480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775327.639838:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664724480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775327.639845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775327.639847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775327.639850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664724480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775328.151188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775328.407229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775329.175148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775329.431225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775330.199172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775330.455233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775331.223201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775331.479276:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775332.247200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775332.503285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775332.759159:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759168:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775332.759172:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775332.759182:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759194:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759200:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775332.759239:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664724608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759263:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759266:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775332.759268:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775332.759271:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759273:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759275:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759283:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775332.759291:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664724736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759297:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759300:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775332.759301:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775332.759305:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759307:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759309:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759313:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775332.759318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664724864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759326:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775332.759328:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775332.759330:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759331:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759334:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775332.759344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664724992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775332.759356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775332.759359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664724608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775332.759360:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759363:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759371:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759376:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775332.759389:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664725120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759398:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759402:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775332.759404:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775332.759408:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759412:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759418:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775332.759432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664725248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775332.759438:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775332.759442:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775332.759447:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775332.759450:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775332.759453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17100 00000100:00080000:3.0:1775775332.759459:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775332.759462:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775332.759466:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775332.759475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664725376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775332.759504:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9dc00 x1862032664724608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759524:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664724608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759535:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759540:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759544:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664724608:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775332.759552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775332.759589:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759601:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9f480 x1862032664724736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759613:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664724736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775332.759642:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775332.759662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759673:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9f100 x1862032664724864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664724864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775332.759734:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775332.759757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9c380 x1862032664724992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664724992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759787:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664724992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775332.759799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775332.759815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9f800 x1862032664725120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664725120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775332.759853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664725248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775332.759870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759880:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9c700 x1862032664725248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664725248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664725248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775332.759908:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664725376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775332.759926:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775332.759935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775332/real 1775775332] req@ffff982980f9e680 x1862032664725376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775332.759945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664725376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775332.759953:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775332.759956:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775332.759958:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664725376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775333.271190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775333.527242:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775334.295202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775334.551235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775335.319173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775335.575246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775336.343163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775336.599226:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775337.367165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775337.623205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775337.879202:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879212:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775337.879216:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775337.879226:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879239:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879245:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775337.879281:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664725504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775337.879309:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879313:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775337.879315:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775337.879319:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879323:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879327:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775337.879339:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775337.879350:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664725504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775337.879353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664725632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775337.879366:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775337.879372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775337.879376:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879380:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879384:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775337.879389:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775337.879400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664725760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775337.879410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879414:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775775337.879417:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775337.879418:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775337.879425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879429:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879433:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775337.879439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775337.879452:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664725888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775337.879456:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2df80 x1862032664725504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775337.879472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879476:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775337.879479:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775337.879483:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664725504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775337.879485:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879491:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879499:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775337.879501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775337.879504:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775337.879508:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879513:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664725504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775337.879519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664726016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775337.879530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775337.879546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775337.879548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775337.879552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725632:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775337.879556:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879563:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879567:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775337.879572:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775337.879585:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775337.879587:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664726144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775337.879599:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775337.879602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2f100 x1862032664725632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775337.879603:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775337.879607:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775337.879614:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17105 00000100:00080000:0.0:1775775337.879621:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775337.879625:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664725632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775337.879631:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775337.879643:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775337.879647:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664726272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775337.879649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879657:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775337.879668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775337.879691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2ca80 x1862032664725760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775337.879719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664725760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775337.879728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775337.879731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775337.879740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775337.879762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2fb80 x1862032664725888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775337.879784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664725888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775337.879791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775337.879794:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664725888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775337.879803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726016:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775337.879821:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2f480 x1862032664726016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775337.879841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664726016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775337.879854:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775337.879857:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775337.879877:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664726144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775337.879897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2d500 x1862032664726144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775337.879919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664726144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775337.879926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775337.879929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664726144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775337.879937:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664726272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775337.879957:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775337.879966:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775337/real 1775775337] req@ffff982974b2d180 x1862032664726272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775337.879977:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664726272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775337.879984:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775337.879987:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775337.879990:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664726272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775338.391169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775338.647297:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775339.415147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775339.671200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775340.439136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775340.695246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775341.463193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775341.719182:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775342.487193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775342.743295:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775342.999222:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775342.999236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999246:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999259:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999265:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775342.999304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664726400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775342.999330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775342.999334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999343:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999352:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775342.999362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664726528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775342.999369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664726400:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775342.999384:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775342.999387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999390:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999392:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775342.999406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664726656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775342.999412:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999415:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775342.999416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999424:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775342.999436:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664726784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775342.999459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775342.999463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999466:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999469:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999471:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999475:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775342.999480:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775342.999485:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664726912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775342.999492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775342.999496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999499:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00080000:3.0:1775775342.999502:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999504:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999507:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775342.999514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664727040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775342.999519:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775342.999522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775342.999523:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775342.999526:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17110 00000100:00100000:0.0:1775775342.999530:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9ed80 x1862032664726400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775342.999531:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775342.999537:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775342.999544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775342.999551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664727168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775342.999557:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664726400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999586:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999591:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999595:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664726400:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775342.999620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775342.999639:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9fb80 x1862032664726528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664726528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775342.999669:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775342.999681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9d500 x1862032664726656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664726656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775342.999706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775342.999717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999723:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9e680 x1862032664726784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664726784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775342.999747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775342.999757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9c700 x1862032664726912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664726912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664726912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775342.999782:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775342.999791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9f800 x1862032664727040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664727040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775342.999818:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775342.999828:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775342.999833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775342/real 1775775342] req@ffff982980f9c380 x1862032664727168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775342.999839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664727168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775342.999844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775342.999846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775342.999848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775343.511167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775343.767207:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775344.535173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775344.791231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775345.559251:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775345.815266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775346.583226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775346.839296:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775347.607190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775347.863276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775348.119162:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775348.119174:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775348.119179:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119203:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119210:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775348.119266:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664727296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775348.119300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775348.119303:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775348.119305:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119312:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119315:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775348.119325:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775348.119338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664727424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775348.119350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775348.119360:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664727296:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775348.119368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775348.119375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119380:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119383:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775348.119391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775348.119407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664727552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775348.119416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775348.119418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775348.119420:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119423:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119428:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119432:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775348.119437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775348.119442:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775348.119453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664727680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775348.119462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775348.119465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775348.119467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119477:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00100000:3.0:1775775348.119479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2c380 x1862032664727296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775348.119482:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119494:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775348.119505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775348.119518:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664727808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775348.119520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664727296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119545:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119553:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775348.119554:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775348.119559:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664727296:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775348.119567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775348.119571:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119577:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119582:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775348.119588:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119590:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727424:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775348.119605:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775348.119617:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775348.119625:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664727936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775348.119636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2e680 x1862032664727424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775348.119662:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775348.119673:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775348.119676:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775348.119681:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17115 00000100:00080000:0.0:1775775348.119685:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775348.119689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664727424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775348.119692:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775348.119706:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775348.119712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775348.119723:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664728064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775348.119732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775348.119767:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2e300 x1862032664727552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775348.119787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664727552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119798:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775348.119803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775348.119815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2d180 x1862032664727680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775348.119828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664727680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119835:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775348.119841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727808:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775348.119852:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2d500 x1862032664727808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775348.119865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664727808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119870:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119872:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664727808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775348.119888:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775348.119898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2f480 x1862032664727936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775348.119911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664727936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664727936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775348.119923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775348.119933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775348.119938:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775348/real 1775775348] req@ffff982974b2fb80 x1862032664728064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775348.119945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664728064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775348.119951:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775348.119953:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775348.119955:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775348.631162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775348.887238:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775349.655208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775349.911246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775350.679153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775350.935253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775351.703226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775351.959282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775352.727136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775352.983239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775353.239165:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239175:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775353.239180:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775353.239191:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239205:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239212:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775353.239248:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664728192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775353.239292:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775353.239297:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775353.239301:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239307:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239315:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775353.239325:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664728320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775353.239335:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775353.239342:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775353.239343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664728192:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775353.239350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239354:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239358:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775353.239379:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664728448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775353.239390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239394:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775353.239396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775353.239401:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239406:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775353.239410:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775353.239414:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775353.239436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664728576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775353.239443:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2ca80 x1862032664728192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775353.239445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775353.239450:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775353.239457:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239463:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239466:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239471:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775353.239474:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664728192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775353.239481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664728704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775353.239486:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775353.239492:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775353.239495:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775353.239497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775353.239500:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775353.239501:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664728192:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775353.239506:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239509:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239512:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239515:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775353.239521:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664728832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775353.239527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728320:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775353.239530:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775353.239533:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775353.239534:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775353.239538:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17120 00000100:00080000:0.0:1775775353.239542:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775353.239544:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775353.239548:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775353.239551:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775353.239560:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664728960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775353.239565:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2f100 x1862032664728320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239580:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664728320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239595:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239598:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775353.239602:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775353.239614:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775353.239619:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2df80 x1862032664728448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664728448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239639:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239642:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775353.239649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775353.239671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775353.239683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2fb80 x1862032664728576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664728576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239704:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239707:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775353.239712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775353.239727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775353.239734:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2f480 x1862032664728704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664728704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239749:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664728704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775353.239764:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775353.239774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775353.239779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2d500 x1862032664728832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664728832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775353.239798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775353.239811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775353.239821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775353/real 1775775353] req@ffff982974b2d180 x1862032664728960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775353.239832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664728960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775353.239841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775353.239844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775353.239847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664728960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775353.751216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775354.007242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775354.775144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775355.031200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775355.799165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775356.055198:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775356.823170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775357.079227:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775357.847162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775358.103275:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775358.359204:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359212:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775358.359217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359226:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359237:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359244:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775358.359276:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664729088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775358.359295:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359298:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775358.359300:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359303:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359305:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359309:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359316:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775358.359327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664729216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775358.359334:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359337:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775358.359338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359341:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359343:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359347:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359350:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775358.359363:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664729344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775358.359370:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664729088:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775358.359373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359375:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775358.359376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359381:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359387:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359400:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775358.359409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664729472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775358.359415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775358.359420:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359424:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359430:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775358.359442:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664729600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775358.359451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775358.359456:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359458:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359461:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359463:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359467:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775358.359475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664729728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775358.359480:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775358.359487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775775358.359489:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775358.359490:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775358.359497:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17125 00000100:00080000:0.0:1775775358.359501:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775358.359503:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775358.359508:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775358.359515:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664729856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775358.359554:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2e300 x1862032664729088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664729088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664729088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775358.359612:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775358.359640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.359663:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2e680 x1862032664729216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664729216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359695:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775358.359744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775358.359768:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.359778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2c380 x1862032664729344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664729344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359805:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359813:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775358.359820:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775358.359845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.359857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2d180 x1862032664729472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664729472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359884:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775358.359898:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775358.359915:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.359923:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2d500 x1862032664729600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359943:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664729600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359951:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359954:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359956:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664729600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775358.359961:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664729728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775358.359973:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.359979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2f480 x1862032664729728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.359987:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664729728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.359992:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.359993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.359996:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664729728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775358.360001:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664729856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775358.360011:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775358.360017:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775358/real 1775775358] req@ffff982974b2fb80 x1862032664729856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775358.360024:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664729856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775358.360029:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775358.360030:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775358.360032:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664729856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775358.871179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775359.127216:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775359.895139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775360.151294:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775360.919265:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775361.175254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775361.943279:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775362.199306:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775362.967206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775363.223208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775363.479179:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479190:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775363.479196:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479209:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479225:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479234:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775363.479280:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664729984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775363.479318:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479321:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775363.479323:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479345:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775363.479358:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664730112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775363.479360:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664729984:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775363.479373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479376:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775363.479377:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479381:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479383:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479387:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479393:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775363.479406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664730240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775363.479416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775363.479421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479431:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479435:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775363.479465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664730368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775363.479476:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775363.479481:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730112:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775363.479483:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775363.479487:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479494:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479499:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479505:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775363.479524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775363.479526:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664730496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775363.479540:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479543:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775363.479545:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775363.479548:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479552:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479555:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479559:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775363.479561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775363.479576:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664730624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775363.479586:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775363.479590:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775363.479593:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775363.479596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730496:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775363.479602:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17130 00000100:00080000:3.0:1775775363.479609:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775363.479613:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775363.479620:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775363.479626:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664730624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775363.479635:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664730752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775363.479661:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664730752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775363.991185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775364.247268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775365.015209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775365.271285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775366.039186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775366.295293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775366.551480:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775366.551543:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775366.551568:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551636:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551647:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551656:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551665:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551676:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775366.551686:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775366.551803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9df80 x1862032664730752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.551838:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664730752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.551854:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.551860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.551867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664730752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775366.551878:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9c000 x1862032664730496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.551894:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664730496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.551903:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.551907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.551912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775366.551920:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9d180 x1862032664730624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.551934:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664730624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.551943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.551947:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.551950:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664730624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775366.551957:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9f480 x1862032664730112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.551970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664730112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.551979:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.551982:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.551987:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775366.551994:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9dc00 x1862032664730240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.552007:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664730240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.552015:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.552019:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.552059:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775366.552122:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9ea00 x1862032664730368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.552141:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664730368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.552150:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.552153:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.552158:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664730368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775366.552165:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775363/real 1775775366] req@ffff982980f9f100 x1862032664729984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775366.552179:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664729984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775366.552188:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775366.552191:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775366.552196:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664729984:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775367.063127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775367.319247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775368.087158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775368.343224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775368.599166:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775368.599177:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775368.599182:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599193:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:3.0:1775775368.599206:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775368.599213:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775368.599249:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664730880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775368.599280:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775368.599285:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775368.599287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599291:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:3.0:1775775368.599296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775368.599300:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775368.599312:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775368.599330:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664731008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775368.599343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664730880:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775368.599344:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775368.599355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775368.599356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599361:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:3.0:1775775368.599365:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775368.599369:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775368.599375:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775368.599391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664731136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775368.599402:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775368.599406:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775368.599408:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599412:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:3.0:1775775368.599417:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775368.599421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775368.599426:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775368.599441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775368.599457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664731264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775368.599462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9ce00 x1862032664730880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775368.599483:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775368.599488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664730880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775368.599490:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775368.599494:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599505:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:0.0:1775775368.599507:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775368.599510:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775368.599515:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775368.599520:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775368.599524:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664730880:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775368.599528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775368.599546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664731392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775368.599557:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775368.599562:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775368.599563:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731008:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775368.599567:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00080000:3.0:1775775368.599580:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775368.599585:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775368.599591:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775368.599607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775368.599608:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664731520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775368.599623:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775368.599627:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9e300 x1862032664731008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775368.599629:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775368.599632:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775368.599645:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17136 00000100:00100000:0.0:1775775368.599653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664731008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775368.599653:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775368.599663:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775368.599670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775368.599672:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775368.599677:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.599682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775368.599690:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664731648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775368.599691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775368.599722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775368.599738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9d880 x1862032664731136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775368.599752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664731136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775368.599761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775368.599765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.599769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775368.599776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775368.599796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775368.599807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9f100 x1862032664731264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775368.599821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664731264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775368.599830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775368.599833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.599837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775368.599844:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731392:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775368.599864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775368.599873:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9ea00 x1862032664731392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775368.599882:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664731392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775368.599889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775368.599893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.599897:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775368.599915:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664731520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775368.599941:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775368.599952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9dc00 x1862032664731520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775368.599964:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664731520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775368.599972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775368.599976:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.599979:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664731520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775368.599985:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664731648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775368.600028:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775368.600039:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775368/real 1775775368] req@ffff982980f9f480 x1862032664731648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775368.600050:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664731648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775368.600059:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775368.600062:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775368.600115:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664731648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775369.111190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775369.367250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775370.135165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775370.391230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775371.159162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775371.415246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775372.183141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775372.439264:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775373.207233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775373.463259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775373.719226:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775373.719234:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775373.719238:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719248:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719259:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719265:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775373.719297:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664731776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775373.719320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775373.719323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775373.719324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775373.719351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664731904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775373.719355:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664731776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775373.719366:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775373.719369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775373.719370:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719373:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719378:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719381:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775373.719391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664732032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775373.719398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775373.719401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775373.719402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719405:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719407:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719410:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775373.719419:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775373.719423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664732160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775373.719431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775373.719434:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775373.719435:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719438:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719443:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775373.719453:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2df80 x1862032664731776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775373.719456:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664732288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775373.719489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775373.719492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664731776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775373.719499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775373.719500:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775373.719502:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775373.719506:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775373.719510:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00100000:3.0:1775775373.719514:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664731776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775373.719515:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719522:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719526:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775373.719533:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664732416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775373.719541:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775373.719541:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731904:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775373.719548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775373.719550:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775373.719553:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17141 00000100:00080000:0.0:1775775373.719556:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775373.719559:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775373.719562:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775373.719566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775373.719577:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664732544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775373.719579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2f100 x1862032664731904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664731904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719611:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664731904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775373.719624:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775373.719642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775373.719653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2ca80 x1862032664732032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664732032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719676:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719679:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719683:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775373.719689:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775373.719708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775373.719718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2fb80 x1862032664732160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664732160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775373.719743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775373.719757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775373.719764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2f480 x1862032664732288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664732288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775373.719792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664732416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775373.719802:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775373.719808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2d500 x1862032664732416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664732416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664732416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775373.719828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664732544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775373.719837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775373.719842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775373/real 1775775373] req@ffff982974b2d180 x1862032664732544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775373.719849:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664732544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775373.719854:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775373.719856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775373.719858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664732544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775374.231161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775374.487252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775375.255179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775375.511275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775376.279139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775376.535263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775377.303174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775377.559250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775378.327126:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775378.583232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775378.839172:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839183:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775378.839189:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775378.839200:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839212:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839219:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775378.839258:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664732672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775378.839289:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839294:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775378.839296:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775378.839300:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839309:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775378.839318:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775378.839333:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664732800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775378.839344:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839349:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775378.839352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775378.839352:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664732672:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775378.839368:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839381:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775378.839386:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775378.839402:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664732928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775378.839418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839423:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775378.839427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775378.839427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775378.839435:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839440:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775378.839451:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775378.839464:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664733056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775378.839469:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2c380 x1862032664732672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775378.839489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839494:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775378.839496:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775378.839500:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664732672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775378.839504:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839512:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775378.839518:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775378.839519:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775378.839526:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:0.0:1775775378.839527:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775378.839532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664732672:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775378.839545:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664733184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775378.839558:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775378.839562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732800:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775378.839566:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775378.839570:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775378.839575:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839580:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839584:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775378.839590:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775378.839603:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664733312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775378.839607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775378.839618:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775378.839622:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775378.839625:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775378.839625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2e680 x1862032664732800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775378.839635:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17146 00000100:00080000:0.0:1775775378.839642:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775378.839648:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775378.839653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664732800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775378.839655:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775378.839673:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664733440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775378.839680:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.839697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.839703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775378.839712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775378.839736:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775378.839748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2e300 x1862032664732928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775378.839761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664732928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775378.839770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.839774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.839778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664732928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775378.839785:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775378.839806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775378.839814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2d180 x1862032664733056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775378.839850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664733056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775378.839856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.839858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.839860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775378.839877:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775378.839898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775378.839910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2d500 x1862032664733184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775378.839923:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664733184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775378.839932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.839935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.839938:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775378.839942:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664733312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775378.839957:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775378.839964:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2f480 x1862032664733312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775378.839973:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664733312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775378.839978:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.839980:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.839982:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664733312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775378.839985:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664733440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775378.839994:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775378.840000:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775378/real 1775775378] req@ffff982974b2fb80 x1862032664733440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775378.840007:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664733440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775378.840012:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775378.840014:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775378.840016:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664733440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775379.351177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775379.607233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775380.375194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775380.631214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775381.399170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775381.655257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775382.423154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775382.679196:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775383.447144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775383.703245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775383.959218:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959229:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775383.959234:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959243:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959254:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775383.959260:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775383.959297:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664733568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775383.959320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775383.959324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775383.959332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775383.959340:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775383.959349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664733696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775383.959356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959358:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775383.959360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959363:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959364:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775383.959368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664733568:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775383.959370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775383.959378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775383.959387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664733824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775383.959398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959402:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775383.959403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959408:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775383.959433:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775383.959440:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775383.959442:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775383.959460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664733952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775383.959471:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775775383.959478:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b6a00 x1862032664733568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775383.959479:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959489:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959496:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775383.959501:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664733568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775383.959503:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775383.959514:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775383.959516:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959521:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959524:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664733568:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775383.959530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664734080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775383.959532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733696:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775383.959546:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959549:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775383.959551:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959556:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:2.0:1775775383.959562:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775383.959567:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775383.959569:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775383.959577:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775383.959587:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b7100 x1862032664733696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775383.959592:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664734208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775383.959618:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775383.959622:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775383.959623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664733696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775383.959625:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775383.959635:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17151 00000100:00080000:3.0:1775775383.959637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775383.959641:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775383.959642:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775383.959650:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733696:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775383.959660:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775383.959674:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664734336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775383.959681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775383.959708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b7b80 x1862032664733824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775383.959726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664733824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775383.959751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775383.959765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775383.959789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b4e00 x1862032664733952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775383.959814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664733952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775383.959830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664733952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775383.959843:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775383.959865:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959877:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b6680 x1862032664734080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775383.959886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664734080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775383.959891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959895:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775383.959899:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664734208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775383.959913:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b4000 x1862032664734208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775383.959927:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664734208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775383.959932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959934:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959936:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664734208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775383.959951:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664734336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775383.959962:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775383.959968:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775383/real 1775775383] req@ffff9829764b5500 x1862032664734336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775383.959975:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664734336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775383.959982:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775383.959985:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775383.959988:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664734336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775384.471173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775384.727258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775385.495159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775385.751264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775386.519201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775386.775281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775387.543223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775387.799280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775388.567173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775388.823250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775389.079169:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079176:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775389.079182:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079192:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079205:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079213:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775389.079252:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664734464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775389.079281:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079285:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775389.079287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079292:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775389.079312:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775389.079317:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664734464:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775389.079327:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664734592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775389.079338:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079342:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775389.079344:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079348:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079355:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775389.079367:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775389.079383:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664734720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775389.079387:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775389.079397:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775389.079403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079408:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079412:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079416:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775389.079422:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775389.079432:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2ca80 x1862032664734464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775389.079435:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664734848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775389.079468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775389.079473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079478:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00100000:3.0:1775775389.079478:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664734464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775389.079482:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775389.079494:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775389.079495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775389.079501:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775389.079507:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664734976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775389.079508:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664734464:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775389.079540:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079543:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775389.079544:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775389.079547:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079549:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079553:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775389.079556:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775389.079557:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775389.079568:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664735104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775389.079585:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775389.079587:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775389.079592:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775389.079594:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775389.079598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2f100 x1862032664734592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775389.079602:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17156 00000100:00080000:0.0:1775775389.079611:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775389.079635:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775389.079653:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775389.079654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664734592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079668:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775389.079672:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664735232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775389.079673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775389.079685:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775389.079707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775389.079717:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2df80 x1862032664734720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775389.079726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664734720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775389.079736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775389.079739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775389.079751:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775389.079757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2fb80 x1862032664734848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775389.079764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664734848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775389.079773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775389.079776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775389.079786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775389.079792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2f480 x1862032664734976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775389.079799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664734976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775389.079807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664734976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775389.079819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664735104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775389.079834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775389.079844:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2d500 x1862032664735104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775389.079856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664735104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079864:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775389.079871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664735104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775389.079878:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664735232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775389.079896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775389.079903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775389/real 1775775389] req@ffff982974b2d180 x1862032664735232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775389.079911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664735232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775389.079916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775389.079918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775389.079920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664735232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775389.591170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775389.847229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775390.615174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775390.871249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775391.639149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775391.895211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775392.663170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775392.919279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775393.687144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775393.943229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775394.199219:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199229:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775394.199233:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199244:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199258:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199266:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775394.199315:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664735360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775394.199348:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775394.199352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199356:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199359:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199363:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775394.199373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775394.199384:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664735488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775394.199390:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664735360:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775394.199394:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199398:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775394.199399:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199403:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199406:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199408:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775394.199411:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775394.199420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664735616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775394.199432:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199435:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775394.199436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199439:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199441:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775394.199447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775394.199457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664735744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775394.199466:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775775394.199474:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775394.199475:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199482:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199488:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199493:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775394.199500:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775394.199513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9d180 x1862032664735360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775394.199516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664735872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775394.199542:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775394.199545:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664735360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775394.199548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775394.199568:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199573:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17161 00000100:00080000:3.0:1775775394.199582:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775394.199587:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775394.199589:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775394.199594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199599:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664735360:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775394.199599:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775394.199618:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664736000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775394.199631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735488:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775394.199631:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775394.199640:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775394.199643:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775394.199657:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17161 00000800:00000100:0.0:1775775394.199661:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775394.199664:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775394.199671:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9c000 x1862032664735488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775394.199677:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775394.199691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664735488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775394.199697:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664736128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775394.199700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775394.199722:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775394.199737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9df80 x1862032664735616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775394.199754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664735616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775394.199760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775394.199768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775394.199780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9c380 x1862032664735744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775394.199793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664735744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775394.199799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775394.199807:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775394.199818:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9f800 x1862032664735872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775394.199830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664735872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775394.199835:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199837:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664735872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775394.199853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775394.199864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9c700 x1862032664736000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775394.199877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664736000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775394.199882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199887:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775394.199890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775394.199901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775394.199906:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775394/real 1775775394] req@ffff982980f9e680 x1862032664736128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775394.199913:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664736128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775394.199918:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775394.199920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775394.199922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775394.711193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775394.967290:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775395.735141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775395.991184:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775396.759207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775397.015271:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775397.783189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775398.039228:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775398.807183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775399.063271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775399.319197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319206:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775399.319210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775399.319218:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:2.0:1775775399.319229:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775399.319235:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775399.319267:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664736256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775399.319303:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319342:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775399.319343:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775399.319347:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:2.0:1775775399.319350:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775399.319354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775399.319362:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775399.319375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664736256:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775399.319379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664736384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775399.319393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775399.319399:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775399.319403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:2.0:1775775399.319408:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775399.319413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775399.319419:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775399.319431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664736512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775399.319438:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319441:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775399.319442:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775399.319448:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775399.319448:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:2.0:1775775399.319456:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775399.319488:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775399.319497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775399.319503:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b5f80 x1862032664736256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775399.319514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664736640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775399.319525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664736256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775399.319528:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775399.319541:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775399.319550:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775399.319550:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:0.0:1775775399.319561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775399.319561:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775399.319568:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664736256:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775399.319571:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775399.319581:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775399.319596:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664736768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775399.319601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736384:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775399.319611:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775399.319615:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775399.319616:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775399.319620:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:2.0:1775775399.319625:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775399.319628:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775399.319634:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775399.319639:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775399.319652:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664736896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775399.319657:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b6d80 x1862032664736384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775399.319676:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775399.319682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664736384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775399.319683:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775399.319687:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775399.319696:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17166 00000100:00080000:0.0:1775775399.319701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775399.319702:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775399.319712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775399.319712:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775399.319721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736384:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775399.319721:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775399.319733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775399.319738:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664737024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775399.319761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775399.319773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b4a80 x1862032664736512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775399.319785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664736512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775399.319793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775399.319797:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775399.319800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775399.319806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775399.319823:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775399.319833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b5180 x1862032664736640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775399.319851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664736640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775399.319859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775399.319862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775399.319866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775399.319883:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775399.319902:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775399.319912:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b5c00 x1862032664736768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775399.319924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664736768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775399.319932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775399.319935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775399.319948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664736768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775399.319954:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775399.319972:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775399.319983:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b4700 x1862032664736896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775399.319995:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664736896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775399.320010:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775399.320013:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775399.320017:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664736896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775399.320023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775399.320041:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775399.320051:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775399/real 1775775399] req@ffff9829764b5f80 x1862032664737024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775399.320062:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664737024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775399.320110:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775399.320114:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775399.320118:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775399.831203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775400.087232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775400.855223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775401.111359:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775401.879238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775402.135241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775402.903213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775403.159275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775403.927207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775404.183304:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775404.439179:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439188:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775404.439193:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775404.439201:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439211:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775404.439247:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664737152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775404.439269:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439272:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775404.439273:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775404.439276:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439278:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439281:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775404.439289:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775404.439299:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664737280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775404.439307:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439309:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775404.439312:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775404.439314:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664737152:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775404.439317:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439323:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439326:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775404.439329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775404.439338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664737408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775404.439349:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439352:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775404.439354:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775404.439357:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439359:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439361:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775404.439365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775404.439387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664737536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775404.439394:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775404.439403:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439408:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775404.439410:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775404.439416:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439421:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439425:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439429:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9d500 x1862032664737152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775404.439433:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775404.439449:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664737152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775404.439452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664737664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775404.439472:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775404.439475:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775404.439478:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775404.439479:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775404.439483:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775404.439486:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664737152:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775404.439490:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439494:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775404.439498:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775404.439502:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775404.439511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775404.439511:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664737792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775404.439525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775404.439528:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775404.439529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775404.439532:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17171 00000100:00080000:3.0:1775775404.439535:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775404.439537:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775404.439542:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775404.439546:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775404.439553:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9fb80 x1862032664737280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775404.439557:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664737920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775404.439577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664737280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439585:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439589:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439591:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775404.439596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775404.439609:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439615:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9ed80 x1862032664737408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775404.439622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664737408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439628:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775404.439636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775404.439647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9e680 x1862032664737536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775404.439660:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664737536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775404.439672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737664:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775404.439681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9c700 x1862032664737664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775404.439694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664737664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664737664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775404.439714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775404.439724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9f800 x1862032664737792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775404.439737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664737792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775404.439749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775404.439758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775404.439763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775404/real 1775775404] req@ffff982980f9c380 x1862032664737920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775404.439771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664737920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775404.439775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775404.439777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775404.439779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664737920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775404.951222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775405.207264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775405.975192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775406.231285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775406.999145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775407.255243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775408.023164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775408.279250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775409.047161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775409.303299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775409.559176:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775409.559184:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775409.559188:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775409.559196:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00080000:3.0:1775775409.559208:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559213:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775409.559246:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664738048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775409.559271:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775409.559274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775409.559275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775409.559278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00080000:3.0:1775775409.559293:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559299:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775409.559309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775409.559324:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664738176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775409.559334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664738048:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775409.559342:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775409.559347:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775409.559349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775409.559354:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00080000:3.0:1775775409.559359:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559364:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775409.559370:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775409.559385:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664738304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775409.559396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775409.559400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775775409.559404:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775409.559404:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775409.559411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00080000:3.0:1775775409.559417:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559422:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775409.559428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775409.559437:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9df80 x1862032664738048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775409.559442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664738432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775409.559461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775409.559462:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664738048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775409.559473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775409.559477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775409.559478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559487:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775409.559487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00100000:0.0:1775775409.559493:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664738048:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775409.559495:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559502:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775409.559508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775409.559515:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775409.559524:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664738560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775409.559535:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775409.559538:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775409.559544:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775409.559547:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9c000 x1862032664738176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775409.559548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775409.559558:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00100000:0.0:1775775409.559563:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664738176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775409.559575:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775409.559577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775409.559589:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738176:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775409.559589:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775409.559599:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775409.559601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775409.559618:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664738688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775409.559630:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775409.559634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775409.559636:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775409.559639:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775409.559650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9d180 x1862032664738304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775409.559652:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17177 00000100:00080000:3.0:1775775409.559660:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775409.559665:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775409.559668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664738304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775409.559674:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775409.559685:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775409.559690:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664738816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775409.559696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775409.559707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775409.559726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775409.559735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9f480 x1862032664738432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775409.559743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664738432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775409.559748:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775409.559752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775409.559764:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775409.559775:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775409.559780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9dc00 x1862032664738560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775409.559787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664738560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775409.559792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559794:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775409.559796:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664738560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775409.559800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664738688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775409.559810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775409.559815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9ea00 x1862032664738688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775409.559822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664738688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775409.559827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775409.559830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664738688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775409.559834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664738816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775409.559843:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775409.559849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775409/real 1775775409] req@ffff982980f9f100 x1862032664738816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775409.559855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664738816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775409.559860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775409.559862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775409.559864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664738816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775410.071190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775410.327251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775411.095137:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775411.351244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775412.119199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775412.375245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775413.143144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775413.399260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775414.167243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775414.423275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775414.679223:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775414.679237:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679245:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679257:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679262:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775414.679299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664738944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775414.679323:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679325:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775414.679327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679330:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679335:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775414.679351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664739072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775414.679358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775414.679362:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679364:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679370:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679373:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775414.679383:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664738944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775414.679386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664739200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775414.679397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775414.679401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679405:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775414.679417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664739328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775414.679423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775414.679427:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679441:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775414.679454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664739456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775414.679463:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775414.679463:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775414.679469:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679474:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679477:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775414.679486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664739584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775414.679492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775414.679495:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775414.679496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775414.679499:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17182 00000100:00080000:1.0:1775775414.679502:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775414.679504:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775414.679508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775414.679509:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe1180 x1862032664738944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775414.679516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664739712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775414.679553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664738944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679572:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679576:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664738944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775414.679585:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775414.679613:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.679638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe1500 x1862032664739072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.679653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664739072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679666:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775414.679696:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775414.679719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.679732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe1c00 x1862032664739200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.679746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664739200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679761:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775414.679767:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775414.679787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.679798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe0e00 x1862032664739328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.679812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664739328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775414.679845:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775414.679866:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.679877:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe0a80 x1862032664739456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.679890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664739456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679905:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679908:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775414.679918:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664739584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775414.679938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.679948:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe1f80 x1862032664739584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.679959:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664739584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.679968:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.679971:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.679975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664739584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775414.679981:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664739712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775414.680001:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775414.680012:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775414/real 1775775414] req@ffff982981fe3b80 x1862032664739712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775414.680024:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664739712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775414.680033:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775414.680036:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775414.680040:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664739712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775415.191156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775415.447193:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775416.215205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775416.471264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775417.239164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775417.495243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775418.263201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775418.519274:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775419.287186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775419.543213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775419.799167:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799175:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775419.799179:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775419.799188:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799202:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799208:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775419.799240:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664739840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775419.799260:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799263:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775419.799264:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775419.799267:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799270:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799272:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775419.799279:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775419.799288:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664739968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775419.799295:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799298:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775419.799300:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775419.799303:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664739840:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775419.799305:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799310:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775419.799316:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775419.799326:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664740096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775419.799332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799335:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775419.799337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775419.799340:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799354:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799361:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775419.799365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775419.799368:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775419.799376:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664740224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775419.799384:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775419.799388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775419.799392:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799395:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799397:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799402:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe0380 x1862032664739840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775419.799402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775419.799414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664740352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775419.799421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799423:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775419.799426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775419.799427:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664739840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775419.799430:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799433:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799439:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775419.799445:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775419.799455:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775775419.799455:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664740480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775419.799467:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775419.799467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775419.799473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775419.799475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775419.799478:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664739840:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775419.799482:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17187 00000100:00080000:1.0:1775775419.799488:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775419.799492:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775419.799497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775419.799509:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775419.799510:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664740608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775419.799535:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe0000 x1862032664739968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799556:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664739968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799561:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799564:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799566:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664739968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775419.799571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775419.799581:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe1880 x1862032664740096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799606:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664740096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799611:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799615:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775419.799619:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775419.799631:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799637:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe2300 x1862032664740224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664740224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799654:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775419.799657:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775419.799667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe2680 x1862032664740352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664740352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775419.799698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664740480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775419.799709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe2a00 x1862032664740480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664740480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799731:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664740480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775419.799742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664740608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775419.799752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775419.799757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775419/real 1775775419] req@ffff982981fe3480 x1862032664740608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775419.799764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664740608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775419.799769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775419.799772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775419.799773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664740608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775420.311207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775420.567254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775421.335197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775421.591208:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775422.359179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775422.615261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775423.383223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775423.639279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775424.407217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775424.663289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775424.919159:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919171:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775424.919175:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919187:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919201:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919209:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775424.919274:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664740736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775424.919312:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919315:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775424.919317:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919322:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919325:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919330:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775424.919341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775424.919355:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664740864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775424.919358:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664740736:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775424.919381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775424.919387:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919393:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919397:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919402:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775424.919408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775424.919424:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664740992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775424.919434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919437:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775424.919439:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919442:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17192 00000800:00000100:0.0:1775775424.919447:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775424.919456:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919466:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775424.919473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775424.919488:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664741120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775424.919500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b5500 x1862032664740736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775424.919500:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919510:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775424.919515:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919525:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919531:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919535:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919539:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664740736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775424.919542:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775424.919554:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775424.919558:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664741248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775424.919560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919568:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664740736:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775424.919576:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775424.919579:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775424.919581:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919585:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919588:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919591:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775424.919594:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775424.919599:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775424.919604:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664741376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775424.919612:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775424.919624:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775424.919626:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775424.919628:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775424.919633:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17192 00000100:00080000:2.0:1775775424.919637:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775424.919642:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b4000 x1862032664740864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775424.919646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775424.919658:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664741504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775424.919665:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664740864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775424.919710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775424.919729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b6680 x1862032664740992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775424.919746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664740992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919755:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664740992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775424.919759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775424.919780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b4e00 x1862032664741120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775424.919794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664741120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775424.919808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775424.919819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b7b80 x1862032664741248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775424.919832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664741248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775424.919854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664741376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775424.919865:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919871:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b7100 x1862032664741376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775424.919878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664741376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919883:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664741376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775424.919890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664741504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775424.919900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775424.919907:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775424/real 1775775424] req@ffff9829764b6a00 x1862032664741504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775424.919919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664741504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775424.919926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775424.919929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775424.919933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664741504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775425.431165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775425.687283:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775426.455161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775426.711261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775427.479177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775427.735255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775428.503169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775428.759230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775429.527183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775429.783253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775430.039150:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775430.039157:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775430.039161:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039169:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039180:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039186:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775430.039218:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664741632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039254:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775430.039258:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775430.039260:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039265:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039269:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039273:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039285:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775430.039303:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664741760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039323:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775430.039331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664741632:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775430.039331:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775430.039341:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039348:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039355:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039361:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775430.039374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664741888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775430.039411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775430.039414:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039419:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039424:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039428:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039434:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775430.039447:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664742016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775430.039458:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775430.039459:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039462:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039467:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039470:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039474:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775430.039486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664742144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039495:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775430.039499:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741760:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775430.039502:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775430.039508:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775430.039512:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039518:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039522:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775430.039536:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664742272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775430.039545:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775430.039548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775430.039550:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775430.039555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741888:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775430.039565:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17197 00000100:00080000:3.0:1775775430.039569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775430.039574:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775430.039580:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775430.039588:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664742400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775430.039600:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775430.039636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775430.039658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664742272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775430.039684:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664742400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775430.551202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775430.807307:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775431.575171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775431.831261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775432.599156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775432.855245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775775433.111383:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1775775433.111431:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775775433.111452:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111500:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111508:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111514:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111520:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111528:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775433.111552:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775433.111642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9ea00 x1862032664742144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664742144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775433.111713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9dc00 x1862032664742272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664742272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664742272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775433.111752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9f480 x1862032664742400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664742400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664742400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775433.111779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9e300 x1862032664741760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664741760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775433.111801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9ce00 x1862032664741888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664741888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664741888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775433.111829:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9f100 x1862032664742016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664742016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111846:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775433.111861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775430/real 1775775433] req@ffff982980f9d880 x1862032664741632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775433.111904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664741632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775433.111912:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775433.111916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775433.111919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664741632:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775433.623193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775433.879252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775434.647162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775434.903221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775435.159267:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159278:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775435.159282:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159292:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159304:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159310:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775435.159355:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664742528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775435.159386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775435.159408:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775435.159419:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664742656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159432:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775435.159434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159438:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159441:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159444:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775435.159449:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775435.159452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664742528:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775435.159461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664742784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159471:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775435.159477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159481:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159484:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159488:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775435.159494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775435.159503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664742912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159513:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159516:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775435.159519:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159523:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159527:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159532:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775435.159536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775775435.159539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775435.159551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664743040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159560:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159563:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775435.159565:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159569:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159574:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775435.159577:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775435.159582:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe3480 x1862032664742528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775435.159585:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775435.159594:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664743168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775435.159612:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775435.159620:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775435.159622:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775435.159626:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17202 00000100:00080000:1.0:1775775435.159630:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775435.159635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664742528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775435.159636:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775435.159643:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775435.159655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775775435.159657:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664743296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775435.159667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.159676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664742528:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775435.159686:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775435.159717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.159732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe2a00 x1862032664742656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.159751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664742656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.159768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.159772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.159775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775435.159801:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775435.159824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.159871:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe2680 x1862032664742784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.159886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664742784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.159902:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.159906:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.159909:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775435.159917:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775435.159941:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.159953:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe2300 x1862032664742912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.159965:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664742912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.159974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.159978:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.159983:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664742912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775435.159990:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775435.160011:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.160025:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe1880 x1862032664743040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.160036:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664743040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.160045:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.160048:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.160051:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775435.160056:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664743168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775435.160122:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.160135:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe0000 x1862032664743168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.160146:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664743168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.160154:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.160158:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.160161:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664743168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775435.160167:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664743296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775435.160187:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775435.160197:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775435/real 1775775435] req@ffff982981fe0380 x1862032664743296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775435.160207:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664743296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775435.160214:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775435.160216:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775435.160219:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664743296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775435.671156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775435.927224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775436.695165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775436.951202:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775437.719158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775437.975229:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775438.743166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775438.999216:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775439.767175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775440.023222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775440.279172:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279185:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775440.279189:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279197:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279209:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279214:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775440.279247:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664743424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775440.279271:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775440.279275:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279283:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775440.279290:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775440.279298:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664743552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775440.279305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775440.279309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279317:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775440.279321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775440.279325:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664743424:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775440.279332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664743680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775440.279344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279356:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775440.279358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279362:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279367:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279372:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775440.279379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775440.279392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664743808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775440.279401:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775440.279405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279408:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775440.279409:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279413:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279417:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775440.279424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775440.279432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664743936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775440.279439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775440.279443:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775440.279445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe3b80 x1862032664743424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775440.279445:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775440.279453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279467:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279473:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664743424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775440.279473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775440.279487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664744064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775440.279496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279505:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775440.279506:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775440.279511:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664743424:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775440.279512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775440.279517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775440.279521:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743552:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775440.279526:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17207 00000100:00080000:1.0:1775775440.279531:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775440.279535:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775440.279540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775440.279550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775440.279553:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664744192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775440.279567:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe1f80 x1862032664743552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279581:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664743552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279588:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279591:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279594:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775440.279613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775440.279627:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe0a80 x1862032664743680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664743680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279650:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279652:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775440.279656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743808:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775440.279666:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279671:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe0e00 x1862032664743808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279678:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664743808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279692:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775440.279695:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775440.279704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe1c00 x1862032664743936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664743936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664743936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775440.279728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775440.279738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe1500 x1862032664744064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664744064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775440.279790:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775440.279804:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775440.279810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775440/real 1775775440] req@ffff982981fe1180 x1862032664744192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775440.279817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664744192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775440.279822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775440.279824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775440.279826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775440.791177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775441.047242:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775441.815157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775442.071233:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775442.839200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775443.095224:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775443.863155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775444.119241:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775444.887277:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775445.143279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775445.399177:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399187:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775445.399191:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399202:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399215:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399222:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775445.399275:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664744320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775445.399305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775445.399309:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399313:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399317:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399320:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399330:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775445.399346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664744448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775445.399348:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664744320:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775445.399365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399369:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775445.399371:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399377:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399382:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775445.399404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664744576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775445.399411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399414:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775445.399416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399419:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399425:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775445.399429:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775445.399441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664744704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775445.399448:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399451:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775445.399453:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399455:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399457:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399460:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775445.399473:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b6a00 x1862032664744320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775445.399473:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664744832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775445.399503:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775445.399507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775445.399510:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664744320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775445.399510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399521:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399526:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775445.399527:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775445.399536:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664744960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775445.399539:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664744320:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775445.399555:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775445.399558:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775445.399559:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775445.399563:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17212 00000100:00080000:2.0:1775775445.399566:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775445.399568:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775445.399570:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775445.399576:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775445.399580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664745088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775445.399600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399612:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b7100 x1862032664744448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399621:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664744448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399629:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775445.399636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775445.399647:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b7b80 x1862032664744576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664744576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775445.399702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775445.399718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399726:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b4e00 x1862032664744704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399734:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664744704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775445.399748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775445.399758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b6680 x1862032664744832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664744832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399796:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664744832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775445.399802:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775445.399824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399836:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b4000 x1862032664744960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664744960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664744960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775445.399889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775445.399913:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775445.399925:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775445/real 1775775445] req@ffff9829764b5500 x1862032664745088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775445.399937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664745088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775445.399943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775445.399946:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775445.399948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775445.911163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775446.167225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775446.935190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775447.191258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775447.959219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775448.215254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775448.983190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775449.239328:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775450.007153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775450.263280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775450.519306:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775450.519319:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519327:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519338:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775450.519344:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775450.519380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664745216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775450.519405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519408:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775450.519409:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519412:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519415:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775450.519417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775450.519426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775450.519442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664745344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775450.519454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775450.519460:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519464:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519469:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775450.519479:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519481:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664745216:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775450.519487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775450.519504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664745472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775450.519516:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519520:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775450.519522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519528:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519533:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775450.519537:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775450.519544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775450.519556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664745600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775450.519563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519566:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775450.519582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519585:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519589:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775450.519596:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775450.519600:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775450.519605:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775450.519617:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664745728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775450.519625:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519627:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775450.519629:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519636:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519639:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775450.519645:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775450.519651:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775450.519654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe1180 x1862032664745216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775450.519666:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664745856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775450.519674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664745216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775450.519686:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775450.519699:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775450.519702:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775450.519717:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17218 00000100:00080000:1.0:1775775450.519725:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775450.519727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775450.519734:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775450.519740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:1.0:1775775450.519745:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775450.519748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664745216:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775450.519763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775450.519769:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664745984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775450.519793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe1500 x1862032664745344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.519817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664745344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.519824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.519828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.519830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775450.519854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775450.519867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519873:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe1c00 x1862032664745472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.519881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664745472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.519887:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.519889:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.519892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775450.519897:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775450.519908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519914:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe0e00 x1862032664745600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.519921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664745600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.519927:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.519929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.519930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775450.519934:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775450.519945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519951:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe0a80 x1862032664745728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.519959:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664745728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.519964:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.519966:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.519967:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664745728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775450.519971:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775450.519981:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.519988:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe1f80 x1862032664745856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.519996:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664745856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.520001:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.520003:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.520004:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775450.520008:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775450.520019:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775450.520025:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775450/real 1775775450] req@ffff982981fe3b80 x1862032664745984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775450.520032:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664745984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775450.520037:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775450.520039:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775450.520041:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664745984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775451.031167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775451.287298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775452.055160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775452.311257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775453.079197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775453.335244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775454.103143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775454.359234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775455.127180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775455.383295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775455.639205:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775455.639221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775455.639231:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:3.0:1775775455.639245:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639252:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775455.639290:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664746112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775455.639326:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639330:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775455.639332:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775455.639337:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:3.0:1775775455.639342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639347:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775455.639357:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775455.639373:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664746112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775455.639374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664746240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775455.639398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639402:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775455.639404:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775455.639409:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:3.0:1775775455.639414:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639418:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775455.639424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775455.639437:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664746368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775455.639444:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775455.639455:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639459:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775455.639461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775455.639467:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:3.0:1775775455.639472:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639477:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775455.639496:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775455.639503:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9d180 x1862032664746112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775455.639511:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664746496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775455.639523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775455.639525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664746112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775455.639534:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775455.639538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775455.639544:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775455.639546:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:0.0:1775775455.639550:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775455.639553:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775455.639555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664746112:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775455.639562:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775455.639567:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775455.639579:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664746624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775455.639580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746240:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775455.639597:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639602:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775455.639604:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775455.639608:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775455.639611:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17223 00000100:00080000:3.0:1775775455.639617:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639621:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9c000 x1862032664746240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775455.639629:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775455.639641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664746240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775455.639648:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664746752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775455.639651:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639658:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775455.639675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746368:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775455.639675:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775455.639683:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775455.639686:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775455.639692:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17223 00000800:00000100:0.0:1775775455.639697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775455.639698:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775455.639705:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9df80 x1862032664746368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775455.639712:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775455.639719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664746368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775455.639728:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664746880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775455.639738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775455.639763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746496:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775455.639778:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9c380 x1862032664746496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775455.639795:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664746496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775455.639802:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775455.639823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746624:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775455.639834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9f800 x1862032664746624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775455.639847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664746624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775455.639852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639857:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664746624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775455.639860:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664746752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775455.639870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639875:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9c700 x1862032664746752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775455.639882:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664746752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775455.639887:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639890:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664746752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775455.639894:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664746880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775455.639903:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775455.639908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775455/real 1775775455] req@ffff982980f9e680 x1862032664746880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775455.639921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664746880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775455.639928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775455.639931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775455.639935:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664746880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775456.151194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775456.407239:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775457.175179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775457.431310:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775458.199168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775458.455236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775459.223186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775459.479277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775460.247220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775460.503242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775460.759150:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775460.759157:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775460.759162:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759169:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759181:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759187:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775460.759232:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664747008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759260:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775460.759263:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775460.759264:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759267:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759270:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759272:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759280:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775460.759290:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664747136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759298:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775460.759303:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775460.759304:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664747008:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775460.759304:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759314:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759316:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759329:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775460.759337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664747264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775460.759346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775460.759347:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759350:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759352:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759354:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775460.759364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664747392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775460.759375:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775460.759377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775460.759379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759404:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759407:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759409:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759413:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775460.759421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664747520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775460.759431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775460.759433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775460.759438:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9ed80 x1862032664747008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775460.759438:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759442:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759447:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775460.759463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664747648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775460.759469:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775460.759473:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664747008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775460.759473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775460.759476:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775460.759482:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17228 00000100:00080000:3.0:1775775460.759485:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775460.759490:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775460.759495:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775460.759500:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775775460.759503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664747776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775460.759508:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759516:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664747008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775460.759541:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775460.759561:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759570:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9fb80 x1862032664747136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759579:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664747136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759590:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759593:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759595:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775460.759600:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775460.759611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9d500 x1862032664747264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664747264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759643:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759646:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775460.759656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747392:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775460.759678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9e680 x1862032664747392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664747392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775460.759721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775460.759736:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9c700 x1862032664747520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664747520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664747520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775460.759767:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664747648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775460.759777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9f800 x1862032664747648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759789:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664747648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759798:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664747648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775460.759801:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664747776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775460.759810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775460.759816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775460/real 1775775460] req@ffff982980f9c380 x1862032664747776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775460.759823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664747776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775460.759827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775460.759829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775460.759831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664747776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775461.271169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775461.527281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775462.295198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775462.551319:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775463.319186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775463.575267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775464.343209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775464.599269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775465.367164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775465.623271:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775465.879217:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879226:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775465.879230:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879238:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879249:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879255:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775465.879288:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664747904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775465.879330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775465.879335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879338:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879341:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879345:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775465.879354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775465.879368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664747904:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775465.879369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664748032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775465.879387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879391:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775465.879393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879398:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879401:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879406:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775465.879411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775465.879432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664748160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775465.879440:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879442:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775465.879444:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879446:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879449:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879454:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775465.879456:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775465.879459:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775465.879471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664748288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775465.879481:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775465.879487:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879492:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879497:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879501:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775465.879507:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2e300 x1862032664747904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775465.879509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775465.879528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664748416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775465.879535:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664747904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775465.879546:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879554:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775465.879556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775465.879557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775465.879562:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:3.0:1775775465.879565:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775465.879570:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775465.879572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664747904:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775465.879578:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775465.879585:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775465.879598:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664748544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775465.879602:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748032:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775465.879619:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775465.879621:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775465.879623:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775465.879627:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17233 00000100:00080000:0.0:1775775465.879630:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775465.879634:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775465.879638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775465.879641:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775465.879653:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664748672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775465.879654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2e680 x1862032664748032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.879688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664748032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.879698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.879702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.879706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775465.879716:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775465.879740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775465.879752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2c380 x1862032664748160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.879766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664748160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.879776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.879779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.879783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775465.879790:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775465.879814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775465.879825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2d180 x1862032664748288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.879839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664748288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.879848:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.879851:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.879855:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775465.879861:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775465.879880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775465.879890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2d500 x1862032664748416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.879902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664748416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.879910:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.879914:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.879917:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775465.879936:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664748544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775465.879956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775465.879967:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2f480 x1862032664748544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.879980:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664748544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.879989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.879992:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.879996:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664748544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775465.880002:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664748672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775465.880022:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775465.880033:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775465/real 1775775465] req@ffff982974b2fb80 x1862032664748672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775465.880046:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664748672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775465.880055:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775465.880058:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775465.880062:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664748672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775466.391180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775466.647249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775467.415152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775467.671249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775468.439205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775468.695297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775469.463171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775469.719335:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775470.487184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775470.743314:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775470.999205:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999229:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775470.999234:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999247:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999264:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999272:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775470.999332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664748800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775470.999360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775470.999364:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999368:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999372:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999375:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999385:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775470.999396:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664748928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775470.999405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775470.999409:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664748800:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775470.999410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999419:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999422:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999424:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999427:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775470.999437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664749056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775470.999444:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999447:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775470.999449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999451:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999453:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999456:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775470.999467:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664749184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775470.999473:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999476:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775470.999478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999481:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999483:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775470.999490:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775470.999500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664749312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775470.999507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999510:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775470.999511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999514:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00080000:0.0:1775775470.999517:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999521:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775470.999535:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664749440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775470.999536:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2df80 x1862032664748800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775470.999556:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775470.999559:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775470.999561:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775470.999565:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17238 00000100:00100000:3.0:1775775470.999569:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664748800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775470.999570:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775470.999576:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775470.999583:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775470.999588:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775470.999592:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664749568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775470.999594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664748800:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775470.999634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775470.999659:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2f100 x1862032664748928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664748928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664748928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775470.999724:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775470.999747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2ca80 x1862032664749056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664749056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775470.999793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775470.999810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999817:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2fb80 x1862032664749184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664749184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775470.999844:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749312:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775470.999854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999859:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2f480 x1862032664749312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999866:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664749312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999875:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775470.999879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664749440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775470.999889:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2d500 x1862032664749440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664749440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999907:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999909:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999911:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664749440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775470.999914:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664749568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775470.999923:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775470.999928:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775470/real 1775775470] req@ffff982974b2d180 x1862032664749568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775470.999937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664749568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775470.999944:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775470.999948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775470.999951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664749568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775471.511157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775471.767281:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775472.535156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775472.791305:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775473.559150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775473.815274:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775474.583203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775474.839286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775475.607176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775475.863297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775476.119214:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119222:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775476.119226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119236:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119254:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119260:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775476.119294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664749696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775476.119321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119324:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775476.119326:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119331:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119334:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119342:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775476.119352:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664749824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775476.119356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664749696:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775476.119365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775476.119369:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119372:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119374:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119376:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119380:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775476.119387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664749952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775476.119393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775476.119398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119400:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119403:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119406:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775476.119417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664750080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775476.119421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775476.119428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119432:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775476.119434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119442:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119445:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119449:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775476.119456:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9df80 x1862032664749696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775476.119458:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664750208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775476.119487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119491:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775476.119492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775476.119496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664749696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775476.119499:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119504:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119509:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775476.119525:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664750336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775476.119526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119543:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664749696:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775476.119552:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775476.119555:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775476.119556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775476.119560:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17243 00000100:00080000:3.0:1775775476.119562:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775476.119564:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775476.119567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775476.119572:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775476.119576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664750464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775476.119597:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9c000 x1862032664749824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664749824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119624:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119626:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775476.119632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775476.119650:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9d180 x1862032664749952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664749952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664749952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775476.119696:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775476.119722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9d880 x1862032664750080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664750080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119743:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775476.119751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775476.119761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9f100 x1862032664750208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664750208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119780:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775476.119785:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664750336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775476.119795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9ce00 x1862032664750336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664750336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664750336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775476.119829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664750464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775476.119839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775476.119845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775476/real 1775775476] req@ffff982980f9e300 x1862032664750464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775476.119859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664750464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775476.119865:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775476.119867:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775476.119868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664750464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775476.631173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775476.887251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775477.655220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775477.911197:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775478.679211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775478.935279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775479.703202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775479.959283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775480.727212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775480.983267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775481.239187:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239196:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775481.239201:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239209:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239220:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239225:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775481.239257:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664750592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775481.239280:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239284:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775481.239285:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239288:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239291:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239293:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239302:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775481.239310:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664750720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775481.239317:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239319:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775481.239320:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239324:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239326:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775481.239335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664750848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775481.239340:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239342:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775481.239343:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239346:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239347:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239349:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239352:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775481.239362:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664750976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775481.239367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775481.239371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239375:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239377:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239379:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775481.239388:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664751104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775481.239395:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664750592:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775481.239398:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775481.239404:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239407:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239409:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239411:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239414:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775481.239421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664751232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775481.239427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775481.239429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775481.239430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775481.239433:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17248 00000100:00080000:1.0:1775775481.239443:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775481.239445:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775481.239448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775481.239458:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664751360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775481.239459:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239493:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe0380 x1862032664750592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239506:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664750592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239516:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239519:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664750592:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775481.239524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775481.239538:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe0000 x1862032664750720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664750720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239559:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239563:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775481.239566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775481.239576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe1880 x1862032664750848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239588:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664750848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239598:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775481.239605:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775481.239615:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe2300 x1862032664750976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664750976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664750976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775481.239655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775481.239666:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe2680 x1862032664751104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664751104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775481.239693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664751232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775481.239702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe2a00 x1862032664751232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239714:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664751232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664751232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775481.239725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664751360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775481.239734:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775481.239739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775481/real 1775775481] req@ffff982981fe3480 x1862032664751360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775481.239746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664751360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775481.239751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775481.239753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775481.239754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664751360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775481.751205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775482.007286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775482.775221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775483.031283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775483.799221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775484.055313:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775484.823198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775485.079270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775485.847210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775486.103237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775486.359241:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359250:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775486.359254:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359263:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359275:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775486.359281:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775486.359315:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664751488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775486.359342:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359345:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775486.359346:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359349:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775486.359354:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359361:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775486.359369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664751616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775486.359375:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359378:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775486.359379:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359381:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359383:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775486.359386:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359389:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775486.359398:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664751744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775486.359406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775775486.359411:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664751488:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775486.359412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359419:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359421:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775486.359423:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359427:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775486.359436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664751872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775486.359443:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359446:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775486.359448:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359451:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359454:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775486.359457:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359462:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775486.359476:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775486.359478:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664752000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775486.359490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359494:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775486.359496:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775486.359501:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359506:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775486.359508:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2c380 x1862032664751488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775486.359515:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359524:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775486.359528:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664751488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775486.359535:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664752128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775486.359536:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359542:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359549:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664751488:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775486.359555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751616:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775486.359567:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775486.359571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775486.359573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775486.359577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775486.359581:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17253 00000100:00080000:0.0:1775775486.359587:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775486.359591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2e680 x1862032664751616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775486.359596:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775486.359607:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775486.359608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664751616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775486.359628:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664752256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775486.359628:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775486.359654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775486.359669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775486.359681:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2e300 x1862032664751744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775486.359693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664751744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775486.359715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775486.359746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775486.359756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2d180 x1862032664751872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775486.359764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664751872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664751872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775486.359778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775486.359789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775486.359795:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2d500 x1862032664752000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775486.359802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664752000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359810:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775486.359814:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664752128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775486.359823:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775486.359828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2f480 x1862032664752128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775486.359835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664752128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664752128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775486.359847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664752256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775486.359857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775486.359862:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775486/real 1775775486] req@ffff982974b2fb80 x1862032664752256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775486.359868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664752256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775486.359873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775486.359874:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775486.359876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664752256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775486.871163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775487.127264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775487.895232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775488.151263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775488.919161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775489.175273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775489.943173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775490.199286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775490.967148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775491.223315:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775491.479182:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775491.479199:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775491.479204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479213:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479224:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479229:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775491.479263:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664752384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775491.479285:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775491.479288:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775491.479290:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479293:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775491.479311:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775491.479326:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664752512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775491.479335:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664752384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775491.479350:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775491.479355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775491.479356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479360:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479365:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479368:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775491.479374:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775491.479387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664752640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775491.479395:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775491.479397:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775491.479398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479401:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479404:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479406:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775491.479410:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775491.479419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664752768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775491.479426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775491.479433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775491.479434:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775491.479439:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479443:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479445:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479449:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775491.479454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775491.479465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664752896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775491.479472:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775491.479474:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775491.479476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479483:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00100000:3.0:1775775491.479484:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2ca80 x1862032664752384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775491.479487:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479497:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775491.479511:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775491.479525:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664753024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775491.479529:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664752384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775491.479583:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775491.479584:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479595:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664752384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775491.479596:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775491.479606:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775491.479613:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17258 00000100:00080000:0.0:1775775491.479619:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775491.479624:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752512:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775491.479630:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775491.479655:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664753152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775491.479667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479691:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2f100 x1862032664752512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.479706:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664752512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.479719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775491.479733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775491.479764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2df80 x1862032664752640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.479795:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664752640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.479808:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775491.479819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775491.479842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2fb80 x1862032664752768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.479878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664752768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479884:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.479886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775491.479893:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775491.479909:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479916:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2f480 x1862032664752896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.479923:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664752896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.479930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664752896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775491.479937:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775491.479948:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.479953:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2d500 x1862032664753024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.479966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664753024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.479971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.479973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.479975:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775491.479991:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775491.480002:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775491.480008:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775491/real 1775775491] req@ffff982974b2d180 x1862032664753152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775491.480016:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664753152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775491.480022:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775491.480024:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775491.480026:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775491.991174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775492.247249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775493.015162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775493.271196:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775494.039178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775494.295257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775495.063179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775495.319227:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775496.087221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775496.343299:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775496.599124:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599133:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775496.599138:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599148:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599161:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599168:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775496.599205:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664753280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775496.599229:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599232:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775496.599235:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599241:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599245:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599251:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775496.599258:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664753280:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775496.599262:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775496.599279:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664753408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775496.599295:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599297:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775496.599301:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599304:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599308:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599311:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775496.599315:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775496.599324:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664753536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775496.599330:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599333:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775496.599334:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599337:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599340:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599343:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775496.599348:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775496.599364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775496.599375:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664753664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775496.599388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599392:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775496.599394:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753536:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775496.599395:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599406:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599412:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599416:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775496.599421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775496.599433:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664753792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775496.599437:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753664:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775496.599451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775496.599455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599459:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599462:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599465:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775496.599469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775496.599479:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664753920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775496.599485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753792:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775496.599490:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775496.599494:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775496.599495:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775496.599499:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17264 00000100:00080000:3.0:1775775496.599501:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775496.599505:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775496.599508:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775496.599518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664754048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775496.599540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775496.599560:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775497.111170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775497.367238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775498.135138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775498.391295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775499.159178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775499.415305:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775499.671402:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775499.671454:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775499.671480:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671521:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671532:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671540:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671549:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671559:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775499.671569:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775499.671600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9f100 x1862032664753920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664753920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664753920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775499.671669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9d880 x1862032664754048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664754048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775499.671708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9ce00 x1862032664753792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664753792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671776:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775499.671784:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9e300 x1862032664753664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664753664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775499.671823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9dc00 x1862032664753408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664753408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775499.671861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9ea00 x1862032664753536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671875:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664753536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664753536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775499.671900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775496/real 1775775499] req@ffff982980f9f480 x1862032664753280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775499.671915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664753280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775499.671923:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775499.671926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775499.671930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664753280:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775500.183193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775500.439243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775501.207186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775501.463303:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775501.719145:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719155:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775501.719159:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719170:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719183:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719191:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775501.719233:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664754176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775501.719259:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719262:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775501.719263:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719267:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719269:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719273:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719282:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775501.719292:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664754304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775501.719300:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719303:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775501.719304:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719307:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719309:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719311:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719314:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775501.719322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664754432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775501.719325:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664754176:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775501.719333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775501.719338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719341:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719343:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719346:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719349:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775501.719358:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664754560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775501.719364:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719367:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775501.719368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719371:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719374:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719376:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775501.719386:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664754688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775501.719391:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719393:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775501.719394:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719397:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719401:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719403:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775501.719412:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775501.719416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664754816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775501.719423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775501.719427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775501.719428:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775501.719431:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17269 00000100:00080000:3.0:1775775501.719436:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775501.719439:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775501.719443:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775501.719451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664754944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775501.719454:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9d180 x1862032664754176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719479:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664754176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719492:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719497:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719502:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664754176:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775501.719510:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775501.719536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719550:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9c000 x1862032664754304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664754304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719578:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719585:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775501.719620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775501.719642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9df80 x1862032664754432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719673:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664754432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775501.719729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775501.719754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9c380 x1862032664754560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664754560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719799:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775501.719805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775501.719828:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9f800 x1862032664754688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664754688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664754688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775501.719872:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775501.719894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9c700 x1862032664754816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719918:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664754816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.719933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775501.719940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775501.719962:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775501.719973:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775501/real 1775775501] req@ffff982980f9e680 x1862032664754944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775501.719986:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664754944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775501.719994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775501.719997:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775501.720001:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664754944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775502.231212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775502.487301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775503.255229:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775503.511302:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775504.279219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775504.535276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775505.303200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775505.559286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775506.327169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775506.583294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775506.839196:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839206:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775506.839210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839222:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839238:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839245:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775506.839303:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664755072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839348:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839351:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775506.839353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839358:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839361:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839365:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775506.839386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664755200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839395:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775506.839396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839399:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839401:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839404:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775506.839414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664755328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775506.839426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839430:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839433:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775506.839450:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664755456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839467:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775506.839468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775506.839469:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664755072:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775506.839474:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839484:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839488:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839492:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775506.839500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664755584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839509:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775506.839511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839514:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17274 00000100:00080000:0.0:1775775506.839516:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839519:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775506.839528:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664755712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775506.839535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775506.839538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775506.839539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775506.839542:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17274 00000800:00000100:3.0:1775775506.839554:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775506.839556:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775506.839561:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775506.839566:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775506.839575:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664755840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775506.839607:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2e300 x1862032664755072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664755072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839650:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664755072:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775506.839659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775506.839677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2e680 x1862032664755200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664755200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775506.839707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775506.839718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839723:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2c380 x1862032664755328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664755328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775506.839762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775506.839773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2d180 x1862032664755456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664755456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839794:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775506.839799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775506.839809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2d500 x1862032664755584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664755584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664755584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775506.839835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664755712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775506.839844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2f480 x1862032664755712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664755712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664755712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775506.839868:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664755840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775506.839878:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775506.839883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775506/real 1775775506] req@ffff982974b2fb80 x1862032664755840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775506.839889:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664755840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775506.839895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775506.839897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775506.839899:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664755840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775507.351190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775507.607313:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775508.375150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775508.631288:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775509.399156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775509.655291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775510.423172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775510.679242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775511.447160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775511.703293:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775511.959234:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959247:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775511.959252:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959262:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959273:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775511.959279:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775511.959315:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664755968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775511.959350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775511.959357:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959363:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959367:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775511.959373:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959385:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775511.959405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664756096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775511.959406:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664755968:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775511.959431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775511.959439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959445:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959449:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775511.959453:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959459:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775511.959469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664756224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775511.959477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959480:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775511.959482:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959485:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959488:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775511.959490:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775511.959521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664756352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775511.959526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775511.959539:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959543:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775511.959546:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959552:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959561:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775511.959566:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959573:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775511.959585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9d500 x1862032664755968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775511.959596:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664756480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775511.959612:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775511.959616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664755968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775511.959621:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775511.959630:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775511.959635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775511.959640:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:0.0:1775775511.959644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775511.959652:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775511.959654:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664755968:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775511.959661:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959669:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775511.959686:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664756608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775511.959688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756096:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775511.959712:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775511.959717:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775511.959719:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775511.959726:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17279 00000100:00080000:3.0:1775775511.959732:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775511.959738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775511.959741:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775511.959748:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775511.959761:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664756736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775511.959764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9fb80 x1862032664756096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.959821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664756096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.959839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.959843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.959846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775511.959856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775511.959875:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775511.959883:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9ed80 x1862032664756224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.959891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664756224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.959898:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.959900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.959902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775511.959906:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775511.959918:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775511.959925:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9e680 x1862032664756352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.959940:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664756352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.959946:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.959948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.959951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775511.959955:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775511.959971:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775511.959981:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9c700 x1862032664756480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.959994:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664756480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.960003:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.960006:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.960010:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775511.960034:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664756608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775511.960051:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775511.960059:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9f800 x1862032664756608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.960091:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664756608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.960097:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.960099:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.960101:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664756608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775511.960106:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664756736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775511.960119:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775511.960126:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775511/real 1775775511] req@ffff982980f9c380 x1862032664756736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775511.960134:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664756736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775511.960139:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775511.960141:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775511.960143:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664756736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775512.471221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775512.727251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775513.495193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775513.751280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775514.519200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775514.775203:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775515.543212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775515.799304:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775516.567210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775516.823272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775517.079172:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775517.079183:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775517.079193:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775517.079202:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079213:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775517.079219:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775517.079252:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664756864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079274:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775517.079277:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775517.079278:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775517.079281:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079284:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775517.079286:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775517.079296:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775517.079305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664756992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079313:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775517.079316:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775517.079318:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775517.079319:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664756864:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775517.079323:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079328:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775517.079332:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775517.079335:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775517.079345:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664757120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079351:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775517.079353:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775517.079355:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775517.079358:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079371:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775517.079374:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775517.079378:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775517.079386:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775517.079387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664757248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775517.079398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775517.079400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775517.079403:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775517.079422:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775517.079426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775517.079434:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe3480 x1862032664756864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775517.079440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664757376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079457:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775517.079460:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664756864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775517.079465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775517.079471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775517.079476:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775517.079478:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:0.0:1775775517.079482:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775517.079486:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775517.079487:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664756864:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775517.079496:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775517.079502:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775517.079511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775517.079517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664757504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775517.079530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775517.079533:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775517.079539:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775517.079542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775517.079546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe2a00 x1862032664756992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775517.079548:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17284 00000100:00080000:1.0:1775775517.079557:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775517.079561:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664756992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775517.079566:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775517.079573:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000008:00100000:1.0:1775775517.079579:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775517.079583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664756992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775517.079590:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775517.079605:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664757632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775517.079607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775517.079616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe2680 x1862032664757120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775517.079630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664757120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775517.079636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775517.079640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775517.079644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775517.079658:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775517.079664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe2300 x1862032664757248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775517.079671:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664757248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775517.079680:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775517.079684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775517.079687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775517.079697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775517.079703:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe1880 x1862032664757376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775517.079709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664757376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775517.079714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775517.079718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775517.079728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664757504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775517.079738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775517.079744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe0000 x1862032664757504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775517.079751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664757504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775517.079756:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775517.079760:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664757504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775517.079763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664757632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775517.079772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775517.079778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775517/real 1775775517] req@ffff982981fe0380 x1862032664757632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775517.079784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664757632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775517.079789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775517.079791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775517.079792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664757632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775517.591190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775517.847298:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775518.615199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775518.871235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775519.639155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775519.895231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775520.663164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775520.919236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775521.687146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775521.943241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775522.199183:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199191:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775522.199195:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199204:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199216:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199223:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775522.199262:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664757760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775522.199288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775522.199292:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199296:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199300:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775522.199311:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775522.199340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664757888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775522.199348:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775522.199353:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664757760:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775522.199353:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775522.199356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199363:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199369:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775522.199374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775522.199383:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664758016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775522.199390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199393:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775522.199394:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199397:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199406:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199410:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775522.199414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775522.199423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664758144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775522.199426:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775522.199434:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199437:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775522.199438:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199444:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199447:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775522.199450:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775522.199456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664758272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775522.199463:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775522.199467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775522.199472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00100000:0.0:1775775522.199473:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe3b80 x1862032664757760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775522.199475:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775522.199481:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775522.199488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775522.199496:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664758400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775522.199501:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664757760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775522.199508:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775522.199515:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775522.199518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775522.199522:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775522.199523:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17289 00000100:00080000:1.0:1775775522.199529:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775522.199531:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775522.199534:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664757760:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775522.199539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775522.199551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664758528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775522.199566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775522.199587:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199596:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe1f80 x1862032664757888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664757888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199610:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199615:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664757888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775522.199618:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775522.199629:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe0a80 x1862032664758016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664758016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199647:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775522.199655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758144:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775522.199667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe0e00 x1862032664758144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199685:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664758144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199694:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775522.199706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758272:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775522.199728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe1c00 x1862032664758272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199749:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664758272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775522.199763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664758400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775522.199777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe1500 x1862032664758400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199790:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664758400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664758400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775522.199803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664758528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775522.199812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775522.199818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775522/real 1775775522] req@ffff982981fe1180 x1862032664758528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775522.199824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664758528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775522.199828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775522.199830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775522.199832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664758528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775522.711197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775522.967272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775523.735155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775523.991259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775524.759239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775525.015320:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775525.783151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775526.039310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775526.807189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775527.063252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775527.319213:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775527.319226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319255:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319266:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319272:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775527.319306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664758656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775527.319331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775527.319336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775527.319358:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775527.319372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664758784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775527.319380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664758656:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775527.319383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775527.319387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319395:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319399:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775527.319404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775527.319412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664758912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775527.319420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775527.319425:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319436:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775527.319441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775527.319443:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775527.319449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664759040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775527.319457:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775527.319463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319466:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319469:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319472:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775527.319476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775527.319478:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9df80 x1862032664758656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775527.319485:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664759168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775527.319493:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775527.319494:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664758656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775527.319497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775527.319499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319503:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319506:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775527.319510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775527.319512:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775527.319513:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775527.319516:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319520:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664758656:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775527.319521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664759296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775527.319530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775527.319533:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775527.319534:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775527.319538:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17294 00000100:00080000:3.0:1775775527.319540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775527.319543:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758784:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775527.319544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775527.319548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775527.319556:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664759424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775527.319565:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319577:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9c000 x1862032664758784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664758784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319599:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319602:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319605:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775527.319611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775527.319628:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9d180 x1862032664758912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664758912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319654:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664758912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775527.319658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775527.319680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9f480 x1862032664759040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664759040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775527.319706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775527.319716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9ea00 x1862032664759168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664759168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775527.319740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664759296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775527.319750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9dc00 x1862032664759296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664759296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664759296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775527.319784:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664759424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775527.319795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775527.319800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775527/real 1775775527] req@ffff982980f9e300 x1862032664759424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775527.319808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664759424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775527.319813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775527.319816:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775527.319819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664759424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775527.831140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775528.087204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775528.855168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775529.111218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775529.879135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775530.135259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775530.903190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775531.159278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775531.927209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775532.183231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775532.439199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439207:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775532.439212:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439232:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775532.439237:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775532.439269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664759552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775532.439293:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439296:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775532.439297:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439300:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439303:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775532.439308:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775532.439318:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775532.439331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664759680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775532.439342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664759552:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775532.439345:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439351:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775532.439353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439358:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439362:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775532.439366:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775532.439372:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775532.439385:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664759808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775532.439394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775532.439398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439401:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439404:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775532.439409:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775532.439409:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775532.439417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775532.439428:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664759936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775532.439434:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775532.439439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439445:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775532.439450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439451:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b5f80 x1862032664759552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775532.439453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775532.439463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664760064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775532.439470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775532.439472:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664759552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775532.439473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775532.439476:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439485:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:0.0:1775775532.439491:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775532.439491:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775532.439499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439504:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664759552:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775532.439509:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775532.439515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775532.439529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664760192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775532.439535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775532.439545:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775532.439550:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775532.439552:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775532.439557:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17299 00000100:00080000:2.0:1775775532.439562:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775532.439575:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775532.439580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775532.439582:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775532.439593:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664760320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775532.439599:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b4700 x1862032664759680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439613:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664759680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775532.439631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775532.439653:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439665:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b5c00 x1862032664759808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664759808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439694:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775532.439700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775532.439717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b5180 x1862032664759936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664759936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664759936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775532.439744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775532.439754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b4a80 x1862032664760064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664760064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775532.439798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664760192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775532.439809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b6d80 x1862032664760192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664760192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664760192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775532.439834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664760320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775532.439843:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775532.439848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775532/real 1775775532] req@ffff9829764b5f80 x1862032664760320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775532.439855:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664760320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775532.439860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775532.439862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775532.439864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664760320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775532.951185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775533.207284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775533.975205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775534.231224:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775534.999218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775535.255281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775536.023203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775536.279208:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775537.047225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775537.303287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775537.559197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559206:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775537.559210:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559234:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559242:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775537.559290:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664760448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775537.559315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775537.559324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559331:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559334:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775537.559342:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775537.559354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664760576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775537.559363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775537.559368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664760448:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775537.559368:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559375:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559378:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559380:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775537.559384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775537.559392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664760704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775537.559399:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775537.559404:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559407:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559411:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559415:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775537.559421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775537.559433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775537.559437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664760832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775537.559450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559455:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775537.559457:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559467:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559473:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9ce00 x1862032664760448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775537.559479:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775537.559494:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664760960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775537.559502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775537.559504:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664760448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775537.559506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775537.559509:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559516:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559523:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775537.559525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775537.559529:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775537.559532:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664760448:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775537.559540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775537.559555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664761088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775537.559565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760576:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775537.559569:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775537.559574:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775537.559576:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775537.559581:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17305 00000100:00080000:3.0:1775775537.559587:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775537.559591:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775537.559597:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775537.559602:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775537.559610:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664761216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775537.559621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9d880 x1862032664760576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664760576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559648:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559652:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775537.559659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775537.559682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9f100 x1862032664760704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664760704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775537.559728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775537.559751:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9e300 x1862032664760832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664760832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775537.559783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760960:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775537.559796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9dc00 x1862032664760960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664760960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664760960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775537.559835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775537.559846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9ea00 x1862032664761088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664761088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775537.559870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775537.559880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775537.559885:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775537/real 1775775537] req@ffff982980f9f480 x1862032664761216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775537.559892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664761216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775537.559896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775537.559898:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775537.559900:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775538.071283:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775538.327254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775539.095195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775539.351232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775540.119132:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775540.375244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775541.143148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775541.399212:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775542.167220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775542.423214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775542.679209:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775542.679221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679229:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679242:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775542.679248:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775542.679284:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664761344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775542.679315:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679319:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775542.679321:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679325:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679329:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775542.679333:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775542.679346:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664761344:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775542.679356:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664761472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775542.679373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679376:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775542.679378:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679381:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679384:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775542.679390:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679393:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775542.679401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664761600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775542.679407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775542.679411:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679414:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679418:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775542.679418:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775542.679423:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679427:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775542.679438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664761728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775542.679445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775542.679450:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679453:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00100000:3.0:1775775542.679456:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2df80 x1862032664761344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775542.679457:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775542.679464:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679472:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775542.679479:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664761344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775542.679481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664761856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775542.679525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775542.679526:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775542.679532:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775542.679533:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775542.679535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775542.679538:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679542:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775542.679542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664761344:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775542.679548:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775542.679558:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664761984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775542.679566:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775542.679567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761472:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775542.679572:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775542.679576:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775542.679579:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17310 00000100:00080000:0.0:1775775542.679582:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775542.679584:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775542.679588:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775542.679595:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775542.679598:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664762112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775542.679610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2f100 x1862032664761472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664761472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679626:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679629:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775542.679637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775542.679649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775542.679655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2ca80 x1862032664761600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679663:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664761600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775542.679677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775542.679689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775542.679694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2fb80 x1862032664761728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664761728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775542.679714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775542.679724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775542.679729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2f480 x1862032664761856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679736:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664761856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664761856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775542.679755:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775542.679765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775542.679771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2d500 x1862032664761984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664761984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679787:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664761984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775542.679790:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664762112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775542.679799:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775542.679804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775542/real 1775775542] req@ffff982974b2d180 x1862032664762112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775542.679811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664762112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775542.679816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775542.679818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775542.679819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664762112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775543.191169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775543.447202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775544.215186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775544.471248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775545.239149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775545.495263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775546.263198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775546.519229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775547.287174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775547.543169:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775547.799172:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775547.799179:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775547.799183:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775547.799192:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799202:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775547.799208:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775547.799239:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664762240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775547.799267:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775547.799281:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775547.799283:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775547.799289:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799294:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775547.799299:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664762240:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775547.799302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775547.799315:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775547.799329:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664762368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775547.799339:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775547.799342:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775547.799343:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775547.799346:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775547.799353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775547.799358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775547.799361:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775547.799369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664762496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775547.799377:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775547.799380:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775547.799381:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775547.799385:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799388:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775547.799391:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799395:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9d180 x1862032664762240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775547.799395:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775547.799405:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664762624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775547.799411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775547.799413:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664762240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775547.799416:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775547.799419:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775547.799428:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:0.0:1775775547.799431:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775547.799431:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775547.799435:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775547.799439:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775775547.799441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775547.799445:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664762240:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775547.799451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664762752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775547.799458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775547.799461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775547.799463:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775547.799469:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762368:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775547.799476:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799482:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775547.799488:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775547.799490:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775547.799497:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775547.799500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9c000 x1862032664762368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775547.799517:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664762880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775547.799518:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664762368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775547.799540:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775547.799543:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775547.799544:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775547.799547:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775547.799551:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775547.799554:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17315 00000100:00080000:3.0:1775775547.799559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775547.799560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762368:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775547.799564:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775547.799571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775547.799574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775547.799582:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664763008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775547.799598:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799607:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9df80 x1862032664762496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775547.799618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664762496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775547.799629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775547.799631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775547.799634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775547.799637:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762624:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775547.799649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9c380 x1862032664762624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775547.799662:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664762624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775547.799667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775547.799669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775547.799671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775547.799674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775547.799684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9f800 x1862032664762752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775547.799696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664762752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775547.799701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775547.799703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775547.799705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664762752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775547.799718:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664762880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775547.799729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799734:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9c700 x1862032664762880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775547.799741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664762880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775547.799746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775547.799748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775547.799750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664762880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775547.799753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775547.799762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775547.799768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775547/real 1775775547] req@ffff982980f9e680 x1862032664763008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775547.799775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664763008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775547.799779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775547.799781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775547.799783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775548.311190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775548.567249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775549.335126:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775549.591227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775550.359164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775550.615209:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775551.383128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775551.639221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775552.407174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775552.663225:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775552.919202:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919210:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775552.919215:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775552.919224:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919235:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775552.919277:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664763136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775552.919298:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919301:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775552.919302:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775552.919305:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919308:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919311:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919320:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775552.919329:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664763264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775552.919336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775552.919341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664763136:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775552.919341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775552.919346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919351:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775552.919364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664763392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775552.919375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775552.919379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775552.919381:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919383:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919386:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919389:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775552.919395:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664763520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775552.919400:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775552.919404:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775552.919413:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775552.919417:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919420:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919426:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775552.919439:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664763648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775552.919446:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919450:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775552.919451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775552.919452:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe1180 x1862032664763136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775552.919455:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775552.919465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775552.919482:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664763776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775552.919489:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664763136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775552.919490:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775552.919498:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775552.919501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775552.919508:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17320 00000100:00080000:1.0:1775775552.919512:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775552.919513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775552.919515:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775552.919522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775552.919524:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919531:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664763136:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775552.919532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664763904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775552.919564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775552.919589:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919601:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe1500 x1862032664763264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919614:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664763264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919626:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919629:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775552.919635:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775552.919652:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919661:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe1c00 x1862032664763392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664763392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775552.919700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775552.919719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe0e00 x1862032664763520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664763520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919748:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775552.919770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775552.919821:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919832:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe0a80 x1862032664763648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664763648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919861:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664763648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775552.919870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775552.919888:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919898:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe1f80 x1862032664763776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919909:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664763776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919921:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775552.919929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775552.919946:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775552.919955:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775552/real 1775775552] req@ffff982981fe3b80 x1862032664763904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775552.919966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664763904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775552.919974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775552.919978:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775552.919981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664763904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775553.431200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775553.687168:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775554.455254:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775554.711220:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775555.479243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775555.735232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775556.503222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775556.759193:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775557.527180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775557.783240:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775558.039138:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039149:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775558.039155:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775558.039165:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039179:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039186:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775558.039237:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664764032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775558.039272:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039275:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775558.039277:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775558.039282:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039285:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039290:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039300:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775558.039313:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664764160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775558.039324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039328:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775558.039331:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775558.039333:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664764032:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775558.039338:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775558.039366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664764288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775558.039376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775558.039382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775558.039386:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039397:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775558.039402:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775558.039420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664764416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775558.039430:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039434:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775558.039436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775558.039441:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe0380 x1862032664764032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775558.039444:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039451:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039458:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775558.039468:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664764032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775775558.039478:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664764544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775558.039484:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039487:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664764032:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775558.039501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039505:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775775558.039508:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764160:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775558.039508:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775558.039517:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039523:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039527:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039532:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775558.039542:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775558.039547:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664764672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775558.039561:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe0000 x1862032664764160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775558.039569:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775558.039577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775558.039579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775558.039584:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17325 00000100:00080000:1.0:1775775558.039587:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775558.039593:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775558.039598:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775558.039605:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664764160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775558.039611:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664764800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775558.039617:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039624:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775558.039640:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775558.039662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775558.039670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe1880 x1862032664764288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775558.039679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664764288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775558.039705:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775558.039727:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775558.039738:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe2300 x1862032664764416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775558.039751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664764416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039768:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775558.039774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775558.039803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775558.039811:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe2680 x1862032664764544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775558.039823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664764544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664764544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775558.039846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664764672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775558.039857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775558.039863:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe2a00 x1862032664764672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775558.039871:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664764672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664764672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775558.039882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664764800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775558.039892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775558.039897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775558/real 1775775558] req@ffff982981fe3480 x1862032664764800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775558.039904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664764800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775558.039923:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775558.039926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775558.039928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664764800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775558.551192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775558.807270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775559.575186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775559.831215:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775560.599173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775560.855309:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775561.623225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775561.879303:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775562.647305:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775562.903245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775563.159151:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159165:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775563.159170:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159183:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159198:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159207:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775563.159255:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664764928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775563.159283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775563.159291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159296:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159300:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159313:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775563.159324:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664764928:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775563.159325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664765056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775563.159339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159342:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775563.159343:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775563.159364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664765184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775563.159374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775563.159379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159384:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159388:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159392:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775563.159414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664765312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775563.159421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775563.159426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159432:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159435:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775563.159449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664765440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775563.159457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765056:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775563.159458:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159464:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775563.159467:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159470:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159472:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159474:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775563.159486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664765568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775563.159493:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775563.159496:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775563.159497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775563.159500:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17330 00000100:00080000:0.0:1775775563.159503:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775563.159505:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775563.159509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775563.159514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775563.159520:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664765696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775563.159549:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775563.159571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775563.159586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664765568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775563.159615:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664765696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775563.671454:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775563.927214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775564.695152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775564.951309:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775565.719248:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775565.975209:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775775566.231295:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775775566.231334:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775775566.231351:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231402:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231415:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231426:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231433:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231440:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775566.231448:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775566.231539:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2d500 x1862032664765440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231590:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664765440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775566.231628:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2f480 x1862032664765568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231643:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664765568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231653:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231660:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664765568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775566.231667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2fb80 x1862032664765696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231680:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664765696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231693:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231697:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664765696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775566.231704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2e680 x1862032664765056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664765056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775566.231746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2e300 x1862032664765184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664765184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775566.231780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2d180 x1862032664765312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664765312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775566.231813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775563/real 1775775566] req@ffff982974b2c380 x1862032664764928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775566.231824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664764928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775566.231837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775566.231840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775566.231843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664764928:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775566.743179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775566.999276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775567.767187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775568.023232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775568.279226:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775568.279234:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775568.279238:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279247:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00080000:2.0:1775775568.279258:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279263:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775568.279294:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664765824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775568.279316:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775568.279319:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775568.279321:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279324:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00080000:2.0:1775775568.279327:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775568.279340:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775568.279354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664765952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775568.279367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775568.279369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664765824:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775568.279372:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775568.279379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279386:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00080000:2.0:1775775568.279391:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775568.279401:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775568.279412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664766080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775568.279422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775568.279426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775568.279428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279432:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00080000:2.0:1775775568.279445:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775568.279456:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775568.279457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775568.279475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664766208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775568.279486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775568.279492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b5500 x1862032664765824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775568.279493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775568.279497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279508:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00100000:3.0:1775775568.279538:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664765824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775568.279544:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775568.279552:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775568.279553:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775568.279560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775775568.279562:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775568.279566:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664765824:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775568.279580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664766336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775568.279592:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775568.279597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765952:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775568.279598:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775568.279603:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775568.279609:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00080000:2.0:1775775568.279615:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279620:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775568.279627:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775568.279633:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.279648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b4000 x1862032664765952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775568.279659:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664766464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775568.279669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664765952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775568.279677:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775568.279681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775568.279686:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775568.279689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775568.279690:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775568.279696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664765952:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775568.279702:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17335 00000100:00100000:3.0:1775775568.279708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766080:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775568.279711:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775568.279719:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775568.279725:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775568.279738:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664766592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775568.279743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.279760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b6680 x1862032664766080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775568.279776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664766080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775568.279786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775568.279790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775568.279794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775568.279800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775568.279823:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.279833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b4e00 x1862032664766208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775568.279844:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664766208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775568.279859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775568.279861:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775568.279864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775568.279870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775568.279890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.279902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b7b80 x1862032664766336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775568.279915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664766336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775568.279923:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775568.279926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775568.279930:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775568.279952:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664766464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775568.279974:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.279986:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b7100 x1862032664766464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775568.280000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664766464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775568.280134:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775568.280140:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775568.280145:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664766464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775568.280153:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664766592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775568.280179:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775568.280191:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775568/real 1775775568] req@ffff9829764b6a00 x1862032664766592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775568.280204:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664766592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775568.280212:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775568.280214:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775568.280218:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664766592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775568.791149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775569.047198:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775569.815171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775570.071237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775570.839176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775571.095214:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775571.863170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775572.119207:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775572.887219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775573.143231:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775573.399250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399262:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775573.399268:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775573.399281:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399297:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775573.399306:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775573.399355:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664766720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775573.399397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775573.399406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775573.399412:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399418:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775573.399443:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775573.399457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775573.399467:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664766720:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775573.399480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664766848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775573.399497:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775573.399506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775573.399512:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399518:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775573.399524:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775573.399533:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775573.399540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775573.399551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664766976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775573.399568:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399573:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775573.399576:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775573.399579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2ca80 x1862032664766720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775573.399588:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399598:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775573.399606:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775573.399609:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664766720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775573.399618:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775573.399623:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.399629:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.399635:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664766720:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775573.399642:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664767104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775573.399659:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775573.399664:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766848:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775573.399669:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775573.399673:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775573.399681:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399689:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775573.399693:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775573.399700:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775573.399710:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775573.399710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2f100 x1862032664766848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775573.399728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664766848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775573.399736:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664767232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775573.399741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.399751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.399760:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775573.399771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766976:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775573.399772:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399781:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775573.399786:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775573.399794:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17340 00000800:00000100:3.0:1775775573.399796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775573.399806:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775573.399811:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2df80 x1862032664766976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775573.399816:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775573.399831:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775573.399835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664766976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775573.399846:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.399854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775573.399855:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664767360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775573.399858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664766976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775573.399871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767104:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775775573.399890:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775573.399895:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775775573.399898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775573.399902:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775573.399913:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2c380 x1862032664767104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775573.399914:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17340 00000100:00080000:0.0:1775775573.399926:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775573.399937:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775573.399944:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664767104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775573.399947:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775573.399959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.399963:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.399966:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775573.399975:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664767488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775573.399985:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775573.400011:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775573.400022:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2d180 x1862032664767232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775573.400038:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664767232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775573.400049:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.400054:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.400058:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775573.400101:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664767360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775573.400131:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775573.400143:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2e300 x1862032664767360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775573.400154:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664767360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775573.400167:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.400170:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.400173:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664767360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775573.400186:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664767488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775573.400202:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775573.400212:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775573/real 1775775573] req@ffff982974b2e680 x1862032664767488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775573.400229:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664767488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775573.400250:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775573.400256:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775573.400261:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664767488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775573.911168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775574.167239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775574.935201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775575.191283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775575.959205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775576.215253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775576.983176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775577.239182:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775578.007149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775578.263248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775578.519196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519207:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775578.519212:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519222:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519235:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775578.519280:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664767616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775578.519305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519308:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775578.519310:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519313:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519315:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519318:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519327:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775578.519349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664767744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775578.519358:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519360:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775578.519362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519365:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519367:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519374:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775578.519380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664767616:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775578.519384:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664767872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775578.519392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519394:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775578.519396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519399:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519401:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775578.519417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664768000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775578.519423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775578.519426:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519429:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519432:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519434:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519437:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775578.519443:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775578.519447:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664768128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775578.519453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775578.519457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519460:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00080000:0.0:1775775578.519463:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519465:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775578.519477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664768256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775578.519480:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2fb80 x1862032664767616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775578.519493:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775578.519500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775578.519501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775578.519506:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17346 00000100:00100000:3.0:1775775578.519509:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664767616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775578.519510:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775578.519517:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775578.519524:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775578.519527:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775578.519532:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664768384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775578.519533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664767616:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775578.519551:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775578.519570:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519579:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2f480 x1862032664767744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664767744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519595:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775578.519616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775578.519626:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519632:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2d500 x1862032664767872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519639:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664767872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519645:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519646:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664767872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775578.519652:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775578.519662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519668:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2e680 x1862032664768000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519675:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664768000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519680:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775578.519687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768128:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775578.519696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2e300 x1862032664768128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664768128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775578.519720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664768256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775578.519729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519734:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2d180 x1862032664768256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664768256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664768256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775578.519753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664768384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775578.519762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775578.519767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775578/real 1775775578] req@ffff982974b2c380 x1862032664768384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775578.519774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664768384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775578.519778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775578.519781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775578.519783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664768384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775579.031174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775579.287256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775580.055182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775580.311249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775581.079191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775581.335237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775582.103151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775582.359233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775583.127172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775583.383295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775583.639210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639219:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775583.639224:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775583.639233:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639245:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639263:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775583.639305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664768512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775583.639328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639332:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775583.639335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775583.639341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775583.639358:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775583.639359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664768512:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775583.639372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664768640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775583.639380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775583.639384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775583.639387:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639392:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775583.639395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775583.639404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664768768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775583.639410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775583.639414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775583.639416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639420:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639423:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775583.639433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775583.639434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775583.639445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664768896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775583.639455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639459:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775583.639461:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775583.639465:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b6a00 x1862032664768512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775583.639469:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639478:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639489:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664768512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775583.639493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775583.639498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639503:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639507:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664768512:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775583.639510:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664769024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775583.639519:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639523:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775583.639524:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775583.639527:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639530:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775583.639531:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768640:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775583.639537:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775583.639541:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775583.639552:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664769152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775583.639559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775583.639568:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775583.639571:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775583.639572:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775583.639574:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b7100 x1862032664768640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775583.639578:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17351 00000100:00080000:2.0:1775775583.639584:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775583.639589:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775583.639593:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775583.639593:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664768640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639603:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775583.639606:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664769280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775583.639609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775583.639627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775583.639642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b7b80 x1862032664768768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775583.639661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664768768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639673:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775583.639693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775583.639717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b4e00 x1862032664768896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775583.639735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664768896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664768896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775583.639749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775583.639762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b6680 x1862032664769024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775583.639774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664769024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775583.639805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664769152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775583.639826:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639837:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b4000 x1862032664769152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775583.639858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664769152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664769152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775583.639882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664769280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775583.639897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775583.639904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775583/real 1775775583] req@ffff9829764b5500 x1862032664769280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775583.639911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664769280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775583.639917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775583.639919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775583.639921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664769280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775584.151148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775584.407210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775585.175190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775585.431294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775586.199166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775586.455330:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775587.223251:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775587.479262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775588.247201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775588.503296:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775588.759224:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759235:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775588.759240:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759252:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759266:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759273:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775588.759336:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664769408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775588.759367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775588.759371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759375:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759378:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759380:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759391:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775588.759401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664769536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775588.759412:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759434:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775588.759438:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759444:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759449:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759457:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759464:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775588.759484:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664769408:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775588.759484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664769664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775588.759512:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775588.759518:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759543:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759547:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759562:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759570:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775588.759585:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664769792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775588.759612:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759617:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775588.759618:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759622:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759626:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775588.759635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775588.759636:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775588.759658:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664769920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775588.759667:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759670:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775588.759673:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759706:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759711:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759715:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759719:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775588.759731:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664770048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775588.759743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9ed80 x1862032664769408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775588.759746:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775588.759760:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775588.759766:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775588.759786:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17356 00000100:00080000:3.0:1775775588.759795:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775588.759798:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775588.759803:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775588.759814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664769408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775588.759815:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664770176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775588.759891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.759905:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.759914:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664769408:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775588.759927:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775588.759956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.759970:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9fb80 x1862032664769536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.759982:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664769536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.759990:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.759993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.759996:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775588.760022:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775588.760037:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.760046:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9d500 x1862032664769664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.760056:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664769664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.760063:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.760066:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.760068:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775588.760074:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769792:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775588.760180:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.760206:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9e680 x1862032664769792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.760217:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664769792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.760225:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.760228:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.760231:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775588.760237:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775588.760256:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.760264:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9c700 x1862032664769920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.760274:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664769920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.760281:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.760285:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.760287:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664769920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775588.760292:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775588.760305:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.760313:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9f800 x1862032664770048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.760329:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664770048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.760335:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.760338:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.760340:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775588.760345:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775588.760358:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775588.760366:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775588/real 1775775588] req@ffff982980f9c380 x1862032664770176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775588.760376:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664770176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775588.760383:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775588.760385:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775588.760388:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775589.271232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775589.527378:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775590.295211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775590.551360:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775591.319227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775591.575254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775592.343184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775592.599237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775593.367173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775593.623258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775593.879207:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879221:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775593.879227:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879238:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879253:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879259:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775593.879293:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664770304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775593.879324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879327:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879330:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775593.879345:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775593.879353:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664770432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879363:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775593.879365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879368:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879370:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879374:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664770304:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775593.879378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775593.879387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664770560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879394:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879397:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775593.879398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879400:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879402:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879404:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775593.879408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775593.879416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664770688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879422:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775593.879427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879429:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879439:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879442:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775593.879445:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775593.879450:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775593.879453:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664770816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775593.879465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775593.879467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879472:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879476:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775593.879481:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775593.879486:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775593.879490:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9df80 x1862032664770304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775593.879501:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664770944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775593.879515:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775593.879521:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664770304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775593.879521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775593.879525:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775593.879534:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17361 00000100:00080000:3.0:1775775593.879540:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775593.879545:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775593.879545:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775593.879554:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775775593.879554:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775593.879560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664770304:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775593.879567:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664771072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775593.879570:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775593.879601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9c000 x1862032664770432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879631:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664770432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775593.879673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775593.879692:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9d180 x1862032664770560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664770560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775593.879737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775593.879750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9f480 x1862032664770688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664770688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775593.879777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775593.879786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9ea00 x1862032664770816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664770816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664770816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775593.879811:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775593.879828:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879834:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9dc00 x1862032664770944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879840:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664770944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664770944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775593.879853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775593.879862:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775593.879868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775593/real 1775775593] req@ffff982980f9e300 x1862032664771072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775593.879874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664771072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775593.879879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775593.879881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775593.879883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775594.391186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775594.647196:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775595.415226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775595.671280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775596.439685:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775596.695311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775597.463145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775597.719247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775598.487207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775598.743303:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775598.999216:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775598.999227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775598.999232:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775598.999243:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999256:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775598.999263:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775598.999301:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664771200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775598.999333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775598.999338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775598.999341:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775598.999347:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999353:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775598.999360:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999361:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664771200:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775598.999373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775598.999401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664771328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775598.999415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775598.999424:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775598.999443:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775598.999445:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775598.999452:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999459:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775598.999465:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775598.999473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775598.999477:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2df80 x1862032664771200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775598.999489:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664771456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775598.999494:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664771200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775598.999519:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775598.999523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775598.999528:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775598.999531:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775775598.999534:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664771200:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775598.999536:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775598.999545:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999551:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775598.999556:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771328:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775598.999563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775598.999580:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664771584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775598.999584:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999596:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2f100 x1862032664771328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999598:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775598.999609:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775598.999614:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775598.999619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664771328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999622:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999632:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775598.999641:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775598.999644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775598.999649:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775598.999649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775598.999652:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775598.999658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775598.999668:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664771712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775598.999675:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2ca80 x1862032664771456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999686:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775598.999697:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775598.999701:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775598.999704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664771456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999709:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:3.0:1775775598.999717:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775598.999718:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775598.999723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775598.999728:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771456:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775598.999739:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775598.999743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775598.999758:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664771840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775598.999770:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775598.999770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775598.999776:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775598.999779:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775598.999783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2df80 x1862032664771584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999786:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17366 00000100:00080000:0.0:1775775598.999794:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775598.999798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664771584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775598.999802:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775598.999813:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775598.999813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775598.999819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775598.999823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775598.999829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775598.999832:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664771968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775598.999845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999855:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2c380 x1862032664771712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775598.999863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664771712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775598.999869:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775598.999871:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775598.999874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664771712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775598.999889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775598.999903:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999910:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2d180 x1862032664771840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775598.999917:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664771840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775598.999922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775598.999924:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775598.999926:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775598.999929:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775598.999938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775598.999944:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775598/real 1775775598] req@ffff982974b2e300 x1862032664771968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775598.999951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664771968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775598.999956:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775598.999958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775598.999960:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664771968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775599.511326:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775599.767225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775600.535189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775600.791262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775601.559141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775601.815249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775602.583192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775602.839267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775603.607272:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775603.863215:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775604.119219:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775604.119230:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775604.119236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119245:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:2.0:1775775604.119265:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119270:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775604.119310:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664772096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775604.119340:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775604.119344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775604.119346:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119351:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:2.0:1775775604.119355:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119359:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775604.119380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664772224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775604.119389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775604.119391:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664772096:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775604.119394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775604.119397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119402:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:2.0:1775775604.119404:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119408:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775604.119420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664772352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775604.119427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775604.119430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775604.119431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119434:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:2.0:1775775604.119445:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119447:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119452:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775604.119455:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775604.119464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664772480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775604.119471:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775604.119474:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775604.119475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119478:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:2.0:1775775604.119481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775604.119496:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b5f80 x1862032664772096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775604.119499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664772608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775604.119524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775604.119527:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664772096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775604.119532:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775604.119537:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775604.119542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775604.119544:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17371 00000100:00080000:3.0:1775775604.119548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775604.119552:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775604.119552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664772096:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775604.119560:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119565:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775604.119578:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664772736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775604.119578:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772224:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775604.119594:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775604.119597:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775604.119598:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775604.119602:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17371 00000800:00000100:3.0:1775775604.119606:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775604.119607:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775604.119612:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775604.119616:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775604.119621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b6d80 x1862032664772224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775604.119628:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664772864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775604.119635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664772224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119651:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775604.119663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775604.119677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775604.119683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b4a80 x1862032664772352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775604.119691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664772352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775604.119704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775604.119715:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775604.119721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b5180 x1862032664772480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775604.119732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664772480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775604.119745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775604.119761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775604.119767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b5c00 x1862032664772608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775604.119774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664772608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664772608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775604.119797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664772736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775604.119807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775604.119813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b4700 x1862032664772736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775604.119820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664772736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664772736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775604.119836:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664772864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775604.119846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775604.119851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775604/real 1775775604] req@ffff9829764b5f80 x1862032664772864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775604.119857:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664772864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775604.119868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775604.119871:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775604.119873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664772864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775604.631235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775604.887261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775605.655138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775605.911265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775606.679189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775606.935281:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775607.703133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775607.959318:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775608.727240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775608.983272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775609.239165:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239191:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775609.239198:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775609.239211:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239236:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775609.239307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664772992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775609.239369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775609.239380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775609.239390:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239398:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239407:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775609.239433:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775609.239456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664773120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775609.239472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239477:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775609.239480:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775609.239485:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239491:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239497:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775609.239506:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664772992:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775609.239508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775609.239529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664773248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775609.239543:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239548:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775609.239564:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775609.239571:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239588:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239593:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775609.239601:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775609.239603:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775609.239644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664773376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775609.239657:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239662:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775609.239667:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775609.239667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b5500 x1862032664772992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775609.239680:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239689:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239698:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775609.239704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664772992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775609.239707:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775609.239735:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.239742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.239754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664772992:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775609.239756:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664773504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775609.239771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773120:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775609.239795:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239855:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775609.239857:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775609.239863:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239878:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775609.239880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775609.239887:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775609.239894:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775609.239904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b4000 x1862032664773120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775609.239910:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664773632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775609.239941:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775609.239944:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775609.239947:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775609.239949:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664773120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775609.239954:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17376 00000100:00080000:2.0:1775775609.239961:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775609.239968:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775609.239971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775775609.239972:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775609.239980:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775609.239987:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664773760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775609.239988:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775609.240023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775609.240045:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775609.240054:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b6680 x1862032664773248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775609.240090:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664773248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775609.240104:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.240106:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.240109:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775609.240114:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775609.240131:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775609.240140:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b4e00 x1862032664773376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775609.240149:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664773376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775609.240155:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.240158:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.240160:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775609.240164:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775609.240197:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775609.240209:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b7b80 x1862032664773504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775609.240222:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664773504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775609.240232:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.240235:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.240240:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664773504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775609.240247:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664773632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775609.240265:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775609.240275:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b7100 x1862032664773632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775609.240289:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664773632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775609.240298:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.240301:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.240304:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664773632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775609.240310:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664773760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775609.240327:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775609.240337:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775609/real 1775775609] req@ffff9829764b6a00 x1862032664773760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775609.240360:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664773760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775609.240369:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775609.240373:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775609.240376:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664773760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775609.751200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775610.007330:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775610.775207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775611.031255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775611.799298:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775612.055220:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775612.823239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775613.079288:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775613.847202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775614.103284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775614.359202:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775614.359235:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775614.359244:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359260:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:2.0:1775775614.359279:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775614.359292:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775614.359349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664773888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775614.359399:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775614.359404:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775614.359406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359413:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:2.0:1775775614.359419:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775614.359426:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775614.359440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775614.359461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664774016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775614.359464:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664773888:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775614.359495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775614.359502:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775614.359505:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359513:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:2.0:1775775614.359522:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775614.359530:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775614.359540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775614.359573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664774144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775614.359576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775614.359602:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775614.359610:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775614.359613:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359633:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:2.0:1775775614.359641:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775614.359643:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b6a00 x1862032664773888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775614.359653:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775614.359672:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775614.359686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664773888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775614.359697:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664774272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775614.359767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775614.359771:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775614.359778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775614.359789:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775775614.359790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664773888:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775614.359799:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359813:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:2.0:1775775614.359822:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775614.359832:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775614.359835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774016:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775614.359851:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775614.359879:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664774400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775614.359887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775614.359908:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775614.359922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b7100 x1862032664774016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775614.359936:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775614.359944:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775614.359959:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00100000:3.0:1775775614.359966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664774016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775614.359970:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775614.359986:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775614.359994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775775614.359999:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775614.360005:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775614.360011:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775614.360023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775614.360030:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664774528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775614.360054:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775614.360058:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360113:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b7b80 x1862032664774144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775614.360134:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664774144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775614.360139:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775614.360147:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775614.360163:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775614.360165:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17381 00000100:00080000:3.0:1775775614.360177:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775614.360178:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775614.360187:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774144:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775614.360192:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360205:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774272:192.168.201.118@tcp:8:kworker.0 00000008:00100000:2.0:1775775614.360208:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775614.360247:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664774656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775614.360259:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360296:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b4e00 x1862032664774272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775614.360325:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664774272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775614.360342:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775614.360349:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775614.360356:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775614.360404:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775614.360449:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360472:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b6680 x1862032664774400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775614.360506:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664774400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775614.360517:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775614.360521:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775614.360525:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775614.360532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664774528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775614.360560:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360573:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b4000 x1862032664774528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775614.360588:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664774528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775614.360597:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775614.360601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775614.360605:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664774528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775614.360611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664774656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775614.360638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775614.360648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775614/real 1775775614] req@ffff9829764b5500 x1862032664774656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775614.360661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664774656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775614.360679:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775614.360683:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775614.360687:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664774656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775614.871195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775615.127277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775615.895256:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775616.151213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775616.919203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775617.175321:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775617.943216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775618.199296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775618.967149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775619.223277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775619.479153:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479161:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775619.479165:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479174:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479186:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775619.479192:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775619.479227:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664774784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775619.479263:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479267:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775619.479268:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479272:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775619.479278:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775619.479286:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775619.479295:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664774912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775619.479302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479305:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775619.479306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479309:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479310:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775619.479313:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775619.479315:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775619.479322:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664775040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775619.479328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479331:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775619.479332:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479335:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479337:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775619.479340:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775619.479344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775619.479354:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664775168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775619.479364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664774784:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775619.479380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775619.479388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479395:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479401:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775619.479407:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775619.479414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775619.479431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664775296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775619.479433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775619.479449:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775619.479456:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479463:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479469:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775619.479470:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b5f80 x1862032664774784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775619.479480:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775619.479490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775619.479493:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664774784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.479513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775619.479519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664775424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775619.479519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.479527:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664774784:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775619.479540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774912:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775619.479542:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775619.479547:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775619.479549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775619.479555:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17386 00000100:00080000:2.0:1775775619.479562:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775619.479564:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775619.479584:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775619.479590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b4700 x1862032664774912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775619.479592:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775619.479607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664774912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775619.479618:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664775552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775619.479719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.479724:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.479743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664774912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775619.479749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775619.479769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775619.479782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b5c00 x1862032664775040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775619.479796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664775040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.479806:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.479810:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.479814:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775619.479847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775619.479875:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775619.479889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b5180 x1862032664775168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775619.479912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664775168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.479921:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.479925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.479929:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775619.479935:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775619.479953:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775619.479971:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b4a80 x1862032664775296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775619.479980:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664775296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.479985:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.479988:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.479990:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775619.479994:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664775424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775619.480005:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775619.480011:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b6d80 x1862032664775424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775619.480019:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664775424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.480034:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.480035:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.480037:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664775424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775619.480041:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664775552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775619.480051:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775619.480058:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775619/real 1775775619] req@ffff9829764b5500 x1862032664775552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775619.480116:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664775552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775619.480127:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775619.480131:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775619.480135:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664775552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775619.991192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775620.247356:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775621.015140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775621.271272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775622.039209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775622.295331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775623.063167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775623.319250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775624.087252:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775624.343252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775624.599218:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599227:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775624.599232:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599241:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599253:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599259:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775624.599301:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664775680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775624.599323:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599326:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775624.599327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599334:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599337:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599340:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599349:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775624.599361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664775808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775624.599364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664775680:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775624.599374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775624.599378:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599384:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599386:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775624.599400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664775936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775624.599408:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775624.599413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599419:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599421:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599425:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775624.599432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664776064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775624.599439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775624.599444:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775624.599445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775624.599449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599453:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599457:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599460:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775624.599475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664776192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775624.599481:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2e680 x1862032664775680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775624.599486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599494:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775624.599498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775624.599503:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599507:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775624.599511:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664775680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775624.599512:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775624.599528:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775624.599548:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664776320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775624.599557:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775624.599560:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775624.599564:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775624.599585:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775624.599587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664775680:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775624.599591:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17392 00000100:00080000:0.0:1775775624.599600:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775624.599604:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775624.599609:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775624.599618:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664776448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775624.599628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775624.599655:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2d500 x1862032664775808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664775808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775624.599710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775624.599723:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2f480 x1862032664775936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664775936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664775936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775624.599753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775624.599765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2fb80 x1862032664776064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664776064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599790:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599792:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775624.599798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775624.599809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2e300 x1862032664776192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664776192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775624.599836:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664776320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775624.599846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2d180 x1862032664776320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599861:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664776320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664776320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775624.599884:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664776448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775624.599895:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775624.599919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775624/real 1775775624] req@ffff982974b2c380 x1862032664776448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775624.599948:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664776448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775624.599959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775624.599963:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775624.599967:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664776448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775625.111150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775625.367251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775626.135189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775626.391239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775627.159135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775627.415250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775628.183181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775628.439280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775629.207146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775629.463195:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775629.719155:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719162:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775629.719167:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719175:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719186:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719199:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775629.719230:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664776576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719252:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775629.719254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719257:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719260:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719262:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719270:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775629.719278:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664776704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775629.719288:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664776576:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775629.719293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775629.719297:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719301:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719303:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719305:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775629.719318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664776832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719327:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775629.719328:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719331:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719333:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719335:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775629.719344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664776960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719351:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719354:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775629.719355:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719364:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719367:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719372:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719376:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775629.719384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664777088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719391:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775629.719396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719399:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775629.719407:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776704:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775629.719408:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719415:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775629.719425:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664777216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775629.719433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775629.719435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775629.719436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775629.719439:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17397 00000100:00080000:1.0:1775775629.719454:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775629.719458:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775629.719462:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775629.719464:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775629.719476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664777344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775629.719501:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775629.719521:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775629.719539:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664777216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775629.719552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664777344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775630.231171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775630.487272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775631.255219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775631.511254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775632.279219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775632.535250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775632.791457:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775632.791502:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775632.791516:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791549:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791555:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791560:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791565:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791569:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775632.791575:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775632.791666:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe0000 x1862032664777216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664777216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664777216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775632.791754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe0380 x1862032664777344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664777344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791779:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791784:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664777344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775632.791798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe2a00 x1862032664776704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664776704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775632.791838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe2680 x1862032664776832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664776832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775632.791876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe2300 x1862032664776960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664776960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791914:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664776960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775632.791931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe1880 x1862032664777088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791946:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664777088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791954:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791962:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775632.791970:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775629/real 1775775632] req@ffff982981fe3480 x1862032664776576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775632.791983:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664776576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775632.791991:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775632.791994:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775632.791997:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664776576:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775633.303343:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775633.559290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775634.327153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775634.583273:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775634.839249:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839261:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775634.839266:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839277:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839290:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839297:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775634.839336:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664777472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775634.839389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775634.839406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839411:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839415:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839420:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775634.839430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775634.839449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664777600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775634.839460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839463:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775634.839465:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839469:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839473:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839477:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775634.839487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775634.839499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664777728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775634.839509:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775634.839514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839519:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839523:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839527:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775634.839533:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775634.839547:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664777856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775634.839555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664777472:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775634.839563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839568:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775634.839570:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839575:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839580:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839584:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775634.839590:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775634.839604:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664777984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775634.839614:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775634.839618:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775634.839621:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839625:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839636:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775634.839637:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775634.839645:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775634.839652:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775634.839667:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664778112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775634.839679:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775634.839680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9f100 x1862032664777472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775634.839685:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775634.839689:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775634.839700:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17402 00000100:00080000:3.0:1775775634.839706:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775634.839712:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775634.839716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664777472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775634.839719:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775634.839729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.839736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.839740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664777472:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775634.839746:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664778240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775634.839751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775634.839787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.839802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9d880 x1862032664777600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.839815:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664777600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.839825:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.839828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.839832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775634.839839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775634.839860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.839871:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9ce00 x1862032664777728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.839883:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664777728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.839891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.839895:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.839901:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775634.839936:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775634.839968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.839985:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9e300 x1862032664777856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.839997:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664777856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.840006:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.840010:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.840013:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775634.840019:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775634.840040:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.840051:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9dc00 x1862032664777984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.840062:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664777984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.840093:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.840097:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.840100:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664777984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775634.840107:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664778112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775634.840131:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.840145:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9ea00 x1862032664778112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.840157:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664778112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.840165:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.840169:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.840172:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664778112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775634.840177:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664778240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775634.840196:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775634.840205:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775634/real 1775775634] req@ffff982980f9f480 x1862032664778240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775634.840216:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664778240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775634.840224:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775634.840227:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775634.840230:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664778240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775635.351195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775635.607327:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775636.375165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775636.631249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775637.399149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775637.655270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775638.423169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775638.679304:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775639.447187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775639.703301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775639.959232:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959240:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775639.959244:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959253:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959277:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959283:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775639.959320:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664778368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959348:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959351:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775639.959352:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959356:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959360:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959365:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959374:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775639.959384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664778496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775639.959401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775639.959404:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664778368:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775639.959405:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959413:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959418:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775639.959435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664778624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775639.959449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959451:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959453:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959456:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775639.959471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664778752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959482:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775639.959483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959486:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959488:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959490:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775639.959498:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775639.959502:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664778880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775639.959512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775639.959515:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959520:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959524:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959527:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775639.959537:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664779008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775639.959545:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775639.959551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2df80 x1862032664778368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775639.959552:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775639.959557:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775639.959565:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17407 00000100:00080000:0.0:1775775639.959569:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775639.959581:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775639.959589:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775639.959597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664778368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775639.959608:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664779136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775639.959613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664778368:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775639.959663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775639.959687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2ca80 x1862032664778496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664778496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775639.959719:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775639.959730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2f100 x1862032664778624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664778624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775639.959756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775639.959768:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2c380 x1862032664778752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664778752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775639.959799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775639.959809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2d180 x1862032664778880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664778880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664778880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775639.959835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775639.959844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2e300 x1862032664779008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664779008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959870:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775639.959874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775639.959883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775639.959889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775639/real 1775775639] req@ffff982974b2fb80 x1862032664779136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775639.959895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664779136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775639.959901:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775639.959903:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775639.959904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775640.471198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775640.727227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775641.495192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775641.751222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775642.519149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775642.775264:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775643.543154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775643.799260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775644.567150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775644.823248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775645.079173:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079205:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775645.079209:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079234:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775645.079290:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664779264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775645.079317:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079320:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775645.079322:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079325:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079328:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079331:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775645.079358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664779392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775645.079363:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664779264:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775645.079368:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775645.079374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079380:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079383:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079386:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775645.079395:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664779520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775645.079403:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079405:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775645.079406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079409:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079412:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079414:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775645.079426:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664779648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775645.079432:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079434:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775645.079436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079439:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079442:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775645.079456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664779776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775645.079458:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775645.079467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775645.079471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079473:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079475:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079478:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775645.079487:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664779904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775645.079497:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775645.079500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775645.079501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775645.079505:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17412 00000100:00080000:0.0:1775775645.079507:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775645.079510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775645.079514:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775645.079514:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2f480 x1862032664779264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775645.079524:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664780032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775645.079540:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664779264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079551:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079558:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079564:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664779264:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775645.079594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775645.079622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2d500 x1862032664779392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079649:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664779392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775645.079673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775645.079694:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2e680 x1862032664779520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664779520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079731:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775645.079736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775645.079753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2fb80 x1862032664779648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079773:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664779648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775645.079795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775645.079816:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2e300 x1862032664779776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664779776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664779776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775645.079872:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775645.079892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079901:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2d180 x1862032664779904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664779904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664779904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775645.079938:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775645.079962:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775645.079969:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775645/real 1775775645] req@ffff982974b2c380 x1862032664780032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775645.079977:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664780032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775645.079983:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775645.079985:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775645.079987:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775645.591208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775645.847250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775646.615127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775646.871304:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775647.639276:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775647.895278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775648.663248:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775648.919276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775649.687250:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775649.943280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775650.199228:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199238:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775650.199242:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775650.199253:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199266:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199273:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775650.199318:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664780160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199343:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199347:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775650.199348:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775650.199352:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199354:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199357:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775650.199365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775650.199376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664780288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199389:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199393:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775650.199396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775650.199396:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664780160:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775650.199404:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199411:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199415:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775650.199422:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775650.199439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664780416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775650.199451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775650.199455:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199458:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199461:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775650.199466:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775650.199473:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664780544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199480:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199483:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775650.199486:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775650.199487:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775650.199491:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199493:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199495:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775650.199499:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775650.199508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664780672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199515:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199517:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775650.199519:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775650.199523:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199526:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199529:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775650.199533:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775650.199533:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe3b80 x1862032664780160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775650.199542:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664780800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775650.199551:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775650.199554:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775650.199556:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775650.199557:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664780160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775650.199563:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17417 00000100:00080000:1.0:1775775650.199569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775650.199575:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775650.199577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775775650.199578:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775650.199586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775775650.199591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664780928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775650.199592:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664780160:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775650.199623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775650.199642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe1f80 x1862032664780288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664780288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775650.199673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775650.199683:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe0a80 x1862032664780416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664780416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775650.199714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775650.199726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe0e00 x1862032664780544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199739:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664780544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199746:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775650.199751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775650.199761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199767:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe1c00 x1862032664780672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664780672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664780672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775650.199791:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775650.199801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe1500 x1862032664780800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664780800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775650.199824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775650.199834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775650.199840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775650/real 1775775650] req@ffff982981fe1180 x1862032664780928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775650.199847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664780928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775650.199851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775650.199853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775650.199856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664780928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775650.711155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775650.967230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775651.735166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775651.991294:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775652.759214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775653.015243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775653.783172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775654.039268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775654.807190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775655.063291:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775655.319171:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319180:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775655.319185:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319196:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319210:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775655.319260:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664781056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775655.319285:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319289:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775655.319290:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319294:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319299:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319309:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775655.319320:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664781184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775655.319331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775655.319334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664781056:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775655.319335:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775655.319337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319351:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319361:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319364:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775655.319382:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664781312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775655.319390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319393:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775655.319395:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319398:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319410:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319413:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319417:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775655.319423:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775655.319427:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664781440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775655.319435:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319438:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775655.319440:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319446:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319450:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319453:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319459:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775655.319465:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b4000 x1862032664781056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775655.319470:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664781568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775655.319480:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319496:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775655.319501:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319513:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319517:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775655.319522:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319526:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775655.319527:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664781056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319543:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775655.319549:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664781696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775655.319552:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664781056:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775655.319580:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775655.319584:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775655.319586:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775655.319592:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17422 00000100:00080000:2.0:1775775655.319597:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775655.319599:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781184:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775655.319606:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775655.319613:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775655.319628:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664781824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775655.319643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b6680 x1862032664781184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319673:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664781184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319679:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775655.319689:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775655.319704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b4e00 x1862032664781312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319718:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664781312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775655.319732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781440:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775655.319743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b7b80 x1862032664781440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664781440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319765:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775655.319769:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775655.319779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b7100 x1862032664781568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664781568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664781568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775655.319811:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664781696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775655.319829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319839:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b6a00 x1862032664781696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664781696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319859:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664781696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775655.319899:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664781824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775655.319920:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775655.319932:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775655/real 1775775655] req@ffff9829764b4000 x1862032664781824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775655.319950:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664781824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775655.319958:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775655.319961:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775655.319965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664781824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775655.831159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775656.087251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775656.855185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775657.111260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775657.879155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775658.135249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775658.903161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775659.159200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775659.927144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775660.183257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775660.439187:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775660.439205:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439219:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439236:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439243:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775660.439289:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664781952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775660.439315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775660.439320:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439323:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439326:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775660.439339:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775660.439349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664782080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775660.439358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775660.439362:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439365:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439367:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439369:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775660.439372:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775660.439382:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664782208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775660.439394:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664781952:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775660.439395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775660.439404:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439412:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439414:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439417:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775660.439422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775660.439432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664782336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775660.439440:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439443:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775660.439444:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439448:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439451:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439454:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775660.439457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775660.439464:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664782464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775660.439471:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775660.439476:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775775660.439487:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775660.439491:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439494:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439497:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775660.439504:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775660.439515:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664782592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775660.439521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775660.439524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775660.439526:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775660.439530:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17427 00000100:00080000:0.0:1775775660.439537:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775660.439541:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439548:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2f100 x1862032664781952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775660.439549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775660.439565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664782720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775660.439575:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664781952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439590:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664781952:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775660.439612:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775660.439631:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439639:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2ca80 x1862032664782080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439648:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664782080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439653:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775660.439686:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775660.439698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439705:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2df80 x1862032664782208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664782208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439719:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439721:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775660.439727:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775660.439739:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439745:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2c380 x1862032664782336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664782336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439766:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775660.439769:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775660.439785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2d180 x1862032664782464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664782464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775660.439812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664782592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775660.439824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2e300 x1862032664782592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664782592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664782592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775660.439850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664782720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775660.439860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775660.439866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775660/real 1775775660] req@ffff982974b2fb80 x1862032664782720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775660.439873:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664782720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775660.439878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775660.439880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775660.439882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664782720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775660.951173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775661.208171:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775661.975158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775662.231250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775662.999179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775663.255256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775664.023135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775664.279245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775665.047178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775665.303247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775665.559192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559204:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775665.559209:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559234:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559242:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775665.559278:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664782848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775665.559305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559309:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775665.559311:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559313:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559316:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559320:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559328:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775665.559336:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664782976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775665.559344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559347:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775665.559348:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559351:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559354:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559356:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775665.559367:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664783104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775665.559367:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664782848:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775665.559381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775665.559384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559387:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559388:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559390:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559399:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775665.559408:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664783232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775665.559415:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775665.559419:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559422:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559424:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559426:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775665.559434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664783360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775665.559441:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559443:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775665.559444:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559446:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559449:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775665.559454:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775665.559456:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775665.559467:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664783488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775665.559473:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775665.559475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775665.559477:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775665.559480:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17433 00000100:00080000:0.0:1775775665.559483:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775665.559485:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775665.559489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775665.559494:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2e680 x1862032664782848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775665.559498:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664783616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775665.559542:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664782848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559552:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559556:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664782848:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775665.559566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775665.559585:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559594:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2d500 x1862032664782976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664782976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664782976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775665.559633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775665.559643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2f480 x1862032664783104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664783104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775665.559670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775665.559681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2fb80 x1862032664783232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664783232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775665.559706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775665.559716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2e300 x1862032664783360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664783360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775665.559740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664783488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775665.559749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2d180 x1862032664783488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664783488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664783488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775665.559773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664783616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775665.559782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775665.559787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775665/real 1775775665] req@ffff982974b2c380 x1862032664783616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775665.559794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664783616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775665.559799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775665.559800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775665.559802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664783616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775666.071159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775666.327269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775667.095247:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775667.351273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775668.119201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775668.375253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775669.143163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775669.399236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775670.167128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775670.423220:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775670.679146:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679156:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775670.679161:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679173:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679187:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679194:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775670.679226:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664783744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679252:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775670.679254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679257:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679264:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679267:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679276:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775670.679285:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664783872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679293:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679296:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775670.679296:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664783744:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775670.679298:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679306:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679308:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679310:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679313:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775670.679322:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664784000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679330:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775670.679332:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679335:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679337:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679339:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775670.679350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664784128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679357:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775670.679361:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679367:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17438 00000800:00000100:3.0:1775775670.679370:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775670.679371:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679375:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679379:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775670.679387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664784256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775670.679395:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775670.679397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679400:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679403:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679405:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679408:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775670.679413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664784384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775670.679419:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775670.679421:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2df80 x1862032664783744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775670.679422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775670.679425:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775670.679430:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17438 00000100:00080000:0.0:1775775670.679433:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775670.679438:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775670.679441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775670.679449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664784512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775670.679458:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664783744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679471:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679474:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679479:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664783744:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775670.679499:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775670.679525:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679535:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2ca80 x1862032664783872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679543:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664783872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679549:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679551:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679553:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664783872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775670.679557:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775670.679568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679574:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2f100 x1862032664784000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679581:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664784000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679586:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679588:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775670.679594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784128:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775670.679605:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2c380 x1862032664784128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664784128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679624:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679625:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679627:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775670.679631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775670.679640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2d180 x1862032664784256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664784256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679660:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775670.679665:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664784384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775670.679675:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2e300 x1862032664784384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679687:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664784384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679697:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664784384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775670.679700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664784512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775670.679708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775670.679713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775670/real 1775775670] req@ffff982974b2fb80 x1862032664784512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775670.679720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664784512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775670.679725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775670.679727:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775670.679742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664784512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775671.191148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775671.447253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775672.215167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775672.471210:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775673.239154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775673.495267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775674.263185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775674.519244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775675.287164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775675.543241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775675.799175:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799184:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775675.799188:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775675.799196:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799207:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799212:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775675.799245:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664784640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775675.799269:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799273:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775675.799274:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775675.799278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799281:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799285:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775675.799295:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775675.799302:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664784640:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775675.799312:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664784768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775675.799324:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799328:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775675.799330:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775675.799334:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775675.799350:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775675.799362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664784896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775675.799373:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799377:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775675.799379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775675.799385:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775675.799389:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799396:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775675.799402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775675.799411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664785024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775675.799418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775675.799424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775675.799425:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9d180 x1862032664784640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775675.799429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799434:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799439:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799453:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664784640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775675.799454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775675.799466:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664785152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775675.799470:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775675.799474:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775675.799479:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775675.799480:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775675.799486:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00100000:0.0:1775775675.799490:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664784640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775675.799491:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799497:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775675.799501:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775675.799506:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664785280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775675.799513:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775675.799515:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775675.799517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775675.799519:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17443 00000100:00080000:3.0:1775775675.799521:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775675.799524:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775675.799528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775675.799529:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775675.799542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664785408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775675.799557:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9c000 x1862032664784768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664784768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799589:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775675.799593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775675.799605:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9df80 x1862032664784896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799621:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664784896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664784896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775675.799655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775675.799679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799693:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9c380 x1862032664785024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664785024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775675.799715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775675.799729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9f800 x1862032664785152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664785152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775675.799756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664785280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775675.799770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9c700 x1862032664785280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664785280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799808:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664785280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775675.799829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664785408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775675.799848:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775675.799859:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775675/real 1775775675] req@ffff982980f9e680 x1862032664785408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775675.799869:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664785408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775675.799877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775675.799880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775675.799883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664785408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775676.311238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775676.567220:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775677.335140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775677.591257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775678.359170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775678.615231:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775679.383151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775679.639245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775680.407185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775680.663262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775680.919223:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919231:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775680.919236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919244:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919255:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919260:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775680.919299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664785536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775680.919332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775680.919339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919363:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664785536:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775680.919364:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775680.919379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664785664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775680.919393:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775680.919397:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919400:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919403:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919406:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775680.919408:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775680.919418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664785792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775680.919425:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775680.919430:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775680.919435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919439:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919441:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775680.919448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775680.919458:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664785920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775680.919461:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2f480 x1862032664785536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775680.919479:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775680.919484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919488:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919492:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775680.919492:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664785536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775680.919499:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775680.919505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775680.919509:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775775680.919515:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664786048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775680.919516:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919522:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664785536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775680.919533:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775680.919537:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919540:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17448 00000100:00080000:0.0:1775775680.919542:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919545:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775680.919549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775680.919572:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775680.919578:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664786176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775680.919586:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775680.919589:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775680.919590:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775680.919595:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17448 00000800:00000100:3.0:1775775680.919600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775680.919602:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775680.919606:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775680.919610:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775680.919620:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664786304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775680.919620:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2d500 x1862032664785664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664785664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919666:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775680.919677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775680.919691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2e680 x1862032664785792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664785792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775680.919718:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775680.919730:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2fb80 x1862032664785920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664785920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919746:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664785920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775680.919754:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775680.919763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2e300 x1862032664786048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664786048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775680.919794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664786176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775680.919804:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2d180 x1862032664786176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664786176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664786176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775680.919831:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664786304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775680.919839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775680.919845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775680/real 1775775680] req@ffff982974b2c380 x1862032664786304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775680.919852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664786304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775680.919857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775680.919858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775680.919860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664786304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775681.431139:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775681.687305:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775682.455180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775682.711276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775683.479178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775683.735262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775684.503202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775684.759258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775685.527149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775685.783195:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775686.039190:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039202:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775686.039208:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039218:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039232:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039241:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775686.039277:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664786432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775686.039301:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039304:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775686.039306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039309:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039313:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039316:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039324:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775686.039334:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664786432:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775686.039335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664786560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775686.039348:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039351:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775686.039352:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039355:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039358:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039360:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775686.039372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664786688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775686.039378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775686.039382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039384:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039389:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039391:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039394:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775686.039400:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775686.039404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664786816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775686.039411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039415:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775686.039416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039419:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039425:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775686.039436:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9d500 x1862032664786432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775686.039438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664786944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775686.039456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775686.039458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039481:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039485:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775686.039488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664786432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775686.039490:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039498:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775686.039504:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039510:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775686.039511:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664787072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775686.039516:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664786432:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775686.039528:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775686.039530:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775686.039532:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775686.039535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17453 00000100:00080000:3.0:1775775686.039537:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775686.039540:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775686.039543:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775686.039546:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775686.039553:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664787200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775686.039575:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9fb80 x1862032664786560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664786560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039620:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039623:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775686.039632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775686.039648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039656:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9ed80 x1862032664786688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664786688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039669:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775686.039678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775686.039688:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9e680 x1862032664786816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664786816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039717:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775686.039726:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775686.039746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9c700 x1862032664786944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664786944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039772:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039776:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664786944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775686.039779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775686.039793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9f800 x1862032664787072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664787072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775686.039829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775686.039839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775686.039845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775686/real 1775775686] req@ffff982980f9c380 x1862032664787200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775686.039852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664787200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775686.039858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775686.039860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775686.039862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775686.551166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775686.807217:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775687.575133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775687.831199:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775688.599159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775688.855232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775689.623214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775689.879246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775690.647180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775690.903258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775691.159184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159193:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775691.159197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159217:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159223:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775691.159276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664787328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775691.159300:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159303:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775691.159305:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159311:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159314:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775691.159322:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775691.159335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664787456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775691.159337:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664787328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775691.159358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775691.159362:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159366:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159386:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775691.159392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775691.159403:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664787584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775691.159428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775691.159433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775691.159434:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775691.159437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159446:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775691.159456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775691.159467:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2f100 x1862032664787328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775691.159469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664787712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775691.159498:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159501:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775691.159503:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775691.159506:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664787328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775691.159510:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159517:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159525:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775691.159526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775691.159528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775691.159533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775691.159539:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664787840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775691.159547:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664787328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775691.159551:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159555:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775691.159557:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159561:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159563:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159566:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775691.159569:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775691.159579:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664787968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775691.159580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787456:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775691.159597:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775691.159601:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775691.159602:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775691.159605:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17458 00000100:00080000:0.0:1775775691.159608:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775691.159611:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775691.159614:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775691.159624:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775691.159625:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664788096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775691.159648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2ca80 x1862032664787456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664787456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775691.159679:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775691.159698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775691.159706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2df80 x1862032664787584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159715:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664787584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775691.159729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775691.159742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775691.159748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2c380 x1862032664787712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664787712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775691.159780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775691.159791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775691.159797:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2d180 x1862032664787840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664787840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159842:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664787840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775691.159856:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775691.159869:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775691.159875:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2e300 x1862032664787968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664787968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159891:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159893:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664787968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775691.159897:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775691.159907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775691.159913:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775691/real 1775775691] req@ffff982974b2fb80 x1862032664788096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775691.159931:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664788096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775691.159944:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775691.159948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775691.159951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775691.671182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775691.927301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775692.695140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775692.951276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775693.719188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775693.975275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775694.743150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775694.999262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775695.767175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775696.023265:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775696.279163:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279171:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775696.279176:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279184:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00080000:2.0:1775775696.279194:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279200:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775696.279234:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664788224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279270:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279275:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775696.279278:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279296:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00100000:3.0:1775775696.279297:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664788224:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775696.279301:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279307:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279319:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775696.279335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664788352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279347:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775696.279351:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279355:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00080000:2.0:1775775696.279358:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279362:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279368:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775696.279380:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664788480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279391:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775696.279398:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279405:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00100000:3.0:1775775696.279405:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788352:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775696.279412:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279418:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775696.279439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664788608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279452:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775696.279452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788480:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775696.279461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775696.279462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279466:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00080000:2.0:1775775696.279470:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279474:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279479:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775696.279488:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:2.0:1775775696.279492:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664788736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279500:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279503:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775696.279505:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279508:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00080000:2.0:1775775696.279511:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775696.279514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788736:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775775696.279519:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279527:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775696.279543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664788864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775696.279560:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775696.279569:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775696.279571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775696.279578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17463 00000100:00100000:3.0:1775775696.279578:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788864:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775696.279587:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775696.279592:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775696.279599:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775696.279613:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664788992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775696.279644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775696.791211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775697.047226:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775697.815186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775698.071275:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775698.839135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775699.095258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775775699.351424:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:3.0:1775775699.351478:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775775699.351499:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351565:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351579:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351592:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351611:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351619:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:3.0:1775775699.351628:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775699.351658:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b5f80 x1862032664788992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664788992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351714:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775699.351722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b4700 x1862032664788864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664788864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664788864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775699.351782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b5c00 x1862032664788736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664788736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351806:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351810:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351813:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775699.351820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b5180 x1862032664788608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351834:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664788608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775699.351855:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b6d80 x1862032664788352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664788352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351876:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351884:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775699.351891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b4a80 x1862032664788480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351905:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664788480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664788480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775699.351933:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775696/real 1775775699] req@ffff9829764b5500 x1862032664788224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775699.351947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664788224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775699.351955:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775699.351958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775699.351963:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664788224:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775699.863184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775700.119338:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775700.887141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775701.143217:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775701.399307:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775701.399323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775701.399329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399344:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399360:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399369:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775701.399410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664789120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775701.399456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775701.399462:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775701.399466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399473:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399481:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399504:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399521:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775701.399543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664789248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775701.399561:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775701.399571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664789120:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775701.399571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775701.399576:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399591:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399613:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399618:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399627:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775701.399648:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664789376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775701.399663:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775701.399679:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775701.399681:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399689:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17468 00000800:00000100:0.0:1775775701.399691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775701.399700:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399708:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399717:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775701.399750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9df80 x1862032664789120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775701.399750:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664789504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775701.399778:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775701.399783:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775701.399785:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399796:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399806:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399812:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399820:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775701.399821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664789120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775701.399843:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664789632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775701.399860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775701.399867:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775701.399873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775701.399876:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775701.399882:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775701.399886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664789120:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775701.399893:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399903:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399910:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399917:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775701.399932:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664789760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775701.399942:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789248:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775701.399953:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775701.399960:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775701.399963:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775701.399969:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17468 00000100:00080000:3.0:1775775701.399976:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775701.399981:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775701.399988:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775701.399994:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775701.400012:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664789888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775701.400024:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9c000 x1862032664789248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664789248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400104:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400112:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400117:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775701.400127:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775701.400161:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775701.400179:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9d180 x1862032664789376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400224:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664789376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400237:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400241:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400245:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775701.400255:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775701.400288:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775701.400305:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9f480 x1862032664789504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400332:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664789504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400365:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400370:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400375:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775701.400385:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775701.400418:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775701.400435:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9ea00 x1862032664789632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400455:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664789632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400468:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400474:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400479:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664789632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775701.400511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664789760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775701.400542:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775701.400558:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9dc00 x1862032664789760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400575:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664789760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400597:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400606:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664789760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775701.400614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664789888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775701.400644:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775701.400659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775701/real 1775775701] req@ffff982980f9e300 x1862032664789888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775701.400675:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664789888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775701.400685:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775701.400690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775701.400694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664789888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775701.911153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775702.167306:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775702.935292:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775703.191308:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775703.959247:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775704.215271:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775704.983179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775705.239369:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775706.007230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775706.263229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775706.519225:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519233:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775706.519238:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519246:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519257:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519262:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775706.519300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664790016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775706.519326:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519329:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775706.519331:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519334:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519336:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519339:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519348:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775706.519357:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664790144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775706.519365:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775706.519370:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519373:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519376:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775706.519391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664790272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775706.519398:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664790016:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775706.519404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775706.519417:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519422:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519425:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519429:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775706.519438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664790400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775706.519443:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519446:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775706.519448:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519451:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519453:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519455:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519459:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775706.519467:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664790528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775706.519473:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775706.519476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519479:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519486:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519488:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519492:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775706.519500:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775706.519502:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664790656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775706.519512:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775706.519515:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775706.519517:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775706.519521:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17474 00000100:00080000:2.0:1775775706.519530:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775706.519535:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775706.519540:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775706.519551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b5500 x1862032664790016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775706.519552:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664790784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775706.519619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664790016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519649:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664790016:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775706.519656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775706.519677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.519687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b4a80 x1862032664790144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.519697:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664790144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519706:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775706.519745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775706.519773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.519787:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b6d80 x1862032664790272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.519801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664790272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775706.519820:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775706.519841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.519854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b5180 x1862032664790400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.519868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664790400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775706.519897:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775706.519921:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.519931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b5c00 x1862032664790528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.519941:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664790528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519950:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519953:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664790528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775706.519958:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664790656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775706.519971:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.519979:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b4700 x1862032664790656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.519987:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664790656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.519994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.519996:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.519999:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664790656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775706.520003:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664790784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775706.520015:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775706.520022:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775706/real 1775775706] req@ffff9829764b5f80 x1862032664790784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775706.520029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664790784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775706.520035:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775706.520038:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775706.520040:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664790784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775707.031202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775707.287188:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775708.055217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775708.311340:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775709.079209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775709.335307:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775710.103148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775710.359221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775711.127137:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775711.383323:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775711.639233:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775711.639253:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639264:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639280:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775711.639289:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775711.639335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664790912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775711.639386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639396:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775711.639400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775711.639427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664791040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639441:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775711.639462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639478:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775711.639488:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664790912:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775711.639489:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639504:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775711.639519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664791168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639560:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639564:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775711.639567:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639571:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639577:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775711.639582:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639589:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775711.639601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775711.639608:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664791296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639621:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639626:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775711.639628:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639633:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639638:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775711.639642:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639648:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775711.639656:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9ce00 x1862032664790912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775711.639663:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664791424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639742:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639747:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775711.639750:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775711.639753:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639758:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775711.639763:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639769:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775711.639787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664790912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775711.639787:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664791552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775711.639841:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775711.639850:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775711.639852:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775711.639856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775711.639858:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17479 00000100:00080000:3.0:1775775711.639872:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775711.639878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775711.639883:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775711.639891:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775711.639895:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664790912:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775711.639906:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664791680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775711.639921:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775711.639995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640046:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9d880 x1862032664791040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640108:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664791040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640120:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640125:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640130:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775711.640163:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775711.640194:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640208:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9f100 x1862032664791168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640224:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664791168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640243:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640247:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640251:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775711.640258:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775711.640278:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640289:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9e300 x1862032664791296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640304:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664791296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640313:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640317:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640321:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775711.640327:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775711.640346:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640357:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9dc00 x1862032664791424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640370:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664791424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640379:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640383:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640387:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775711.640394:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664791552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775711.640411:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640421:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9ea00 x1862032664791552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640434:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664791552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640443:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640446:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640450:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664791552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775711.640456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664791680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775711.640473:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775711.640482:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775711/real 1775775711] req@ffff982980f9f480 x1862032664791680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775711.640496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664791680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775711.640505:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775711.640509:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775711.640512:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664791680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775712.151142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775712.407242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775713.175136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775713.431219:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775714.199152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775714.455205:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775715.223209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775715.479272:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775716.247141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775716.503197:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775716.759135:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759146:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775716.759150:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759160:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759173:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759180:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775716.759223:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664791808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775716.759249:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759252:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775716.759254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759257:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759260:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759262:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759271:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775716.759281:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664791936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775716.759288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775716.759291:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759294:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759296:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759298:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759302:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775716.759309:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664792064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775716.759315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775716.759320:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759329:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00100000:0.0:1775775716.759330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664791808:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775716.759333:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759340:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775716.759362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664792192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775716.759374:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775716.759381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759385:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759389:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759392:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775716.759406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664792320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775716.759412:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775716.759420:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759423:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775716.759424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775716.759428:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759439:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759442:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759446:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775716.759453:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9d180 x1862032664791808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775716.759455:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664792448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775716.759478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775716.759480:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775716.759482:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775716.759485:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664791808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775716.759486:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17484 00000100:00080000:3.0:1775775716.759491:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775716.759498:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775716.759504:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775716.759506:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759513:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775716.759514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664792576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775716.759519:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664791808:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775716.759531:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775716.759549:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759558:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9c000 x1862032664791936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664791936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759578:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664791936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775716.759582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775716.759593:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9df80 x1862032664792064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664792064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759612:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759614:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775716.759644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775716.759665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9c380 x1862032664792192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664792192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775716.759713:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775716.759728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9f800 x1862032664792320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664792320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759756:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775716.759761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664792448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775716.759771:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9c700 x1862032664792448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664792448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664792448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775716.759796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664792576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775716.759806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775716.759811:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775716/real 1775775716] req@ffff982980f9e680 x1862032664792576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775716.759818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664792576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775716.759823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775716.759825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775716.759828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664792576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775717.271188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775717.527315:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775718.295195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775718.551338:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775719.319176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775719.575211:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775720.343137:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775720.599252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775721.367192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775721.623236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775721.879236:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775721.879248:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879256:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879267:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879273:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775721.879307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664792704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775721.879334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775721.879339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879342:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879345:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879347:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775721.879355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775721.879364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664792832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775721.879371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775721.879375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879380:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879382:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775721.879387:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775721.879391:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664792704:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775721.879396:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664792960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775721.879404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775721.879408:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879413:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879417:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775721.879423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775721.879437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664793088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775721.879447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879452:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775775721.879453:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775721.879455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879467:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879472:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775721.879478:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775721.879490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664793216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775721.879492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9ed80 x1862032664792704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775721.879507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775721.879514:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775721.879516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879519:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00080000:3.0:1775775721.879522:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879526:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879530:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664792704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775721.879531:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775721.879540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664793344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775721.879543:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879552:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775721.879553:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775721.879557:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664792704:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775721.879558:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775721.879561:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775721.879566:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17489 00000100:00100000:0.0:1775775721.879569:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792832:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775721.879570:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775721.879577:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775721.879581:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775721.879590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664793472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775721.879599:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879613:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9fb80 x1862032664792832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664792832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879633:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775721.879654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775721.879667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9d500 x1862032664792960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664792960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879687:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879691:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664792960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775721.879695:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775721.879706:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879712:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9e680 x1862032664793088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664793088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775721.879732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775721.879742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9c700 x1862032664793216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664793216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775721.879767:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664793344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775721.879777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9f800 x1862032664793344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879789:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664793344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664793344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775721.879800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664793472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775721.879809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775721.879814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775721/real 1775775721] req@ffff982980f9c380 x1862032664793472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775721.879820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664793472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775721.879826:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775721.879828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775721.879830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664793472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775722.391130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775722.647185:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775723.415174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775723.671288:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775724.439162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775724.695261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775725.463180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775725.719199:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775726.487172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775726.743393:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775726.999209:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999218:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775726.999223:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999231:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999247:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775726.999282:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664793600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775726.999310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999313:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775726.999314:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999322:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999329:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775726.999337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664793728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775726.999343:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775726.999347:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999349:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999351:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999353:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775726.999361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664793856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775726.999371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775726.999375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999378:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999380:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999382:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999385:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775726.999393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664793984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775726.999398:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664793600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775726.999403:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999406:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775726.999407:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999411:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999413:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999415:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775726.999426:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664794112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775726.999433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775726.999436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999438:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999442:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999444:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775726.999453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664794240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775726.999459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775726.999462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1775775726.999465:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775726.999465:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775726.999471:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17494 00000100:00080000:2.0:1775775726.999474:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775726.999476:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775726.999479:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775726.999486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664794368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775726.999510:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b4000 x1862032664793600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664793600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999533:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664793600:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775726.999543:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775726.999558:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b6a00 x1862032664793728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664793728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999583:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775726.999591:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775726.999601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b7100 x1862032664793856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999613:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664793856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999623:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775726.999643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775726.999654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999660:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b7b80 x1862032664793984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664793984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999673:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999675:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999678:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664793984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775726.999681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775726.999691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b4e00 x1862032664794112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999703:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664794112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775726.999714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664794240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775726.999723:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b6680 x1862032664794240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999736:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664794240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664794240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775726.999762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664794368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775726.999777:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775726.999785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775726/real 1775775726] req@ffff9829764b5f80 x1862032664794368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775726.999793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664794368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775726.999798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775726.999800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775726.999803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664794368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775727.511169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775727.767286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775728.535215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775728.791242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775729.559194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775729.815303:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775730.583193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775730.839275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775731.607156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775731.863245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775732.119219:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775732.119232:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119242:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119254:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119260:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775732.119296:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664794496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775732.119320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775732.119325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119333:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775732.119349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664794624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775732.119357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775732.119361:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119364:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119367:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119369:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119373:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775732.119376:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664794496:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775732.119383:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664794752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775732.119395:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775732.119401:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119416:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119447:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119453:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119460:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775732.119463:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775732.119477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664794880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775732.119489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119493:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775732.119495:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775732.119501:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe1180 x1862032664794496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775732.119501:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119508:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119516:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119522:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775732.119526:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664794496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119537:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775775732.119541:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664795008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775732.119544:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119551:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664794496:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775732.119564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794624:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775732.119565:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775732.119570:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775732.119573:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119584:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775732.119588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775732.119591:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119598:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775732.119602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe1500 x1862032664794624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775732.119614:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664795136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775732.119615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664794624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119645:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775732.119647:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775732.119649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794624:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775732.119656:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775732.119670:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775732.119677:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17499 00000100:00080000:1.0:1775775732.119681:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775732.119684:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775732.119688:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775732.119697:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775732.119699:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664795264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775732.119726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775732.119736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe1c00 x1862032664794752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775732.119749:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664794752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119757:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775732.119761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775732.119771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775732.119794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775732.119806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe0e00 x1862032664794880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775732.119818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664794880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775732.119837:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664794880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775732.119846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775732.119866:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775732.119876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe0a80 x1862032664795008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775732.119888:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664795008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775732.119899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775732.119908:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664795136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775732.119927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775732.119938:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe1f80 x1862032664795136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775732.119951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664795136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.119959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775732.119962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.119966:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664795136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775732.119983:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664795264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775732.120004:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775732.120014:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775732/real 1775775732] req@ffff982981fe3b80 x1862032664795264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775732.120026:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664795264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775732.120035:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775732.120038:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775732.120042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664795264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775732.631239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775732.887263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775733.655203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775733.911268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775734.679174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775734.935286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775735.703184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775735.959294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775736.727180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775736.983256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775737.239120:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239128:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775737.239132:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239141:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239152:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239158:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775737.239190:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664795392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775737.239217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239220:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239222:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239225:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775737.239234:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775737.239242:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664795520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239251:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239255:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775737.239257:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239261:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239265:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239272:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239274:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664795392:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775737.239278:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775737.239294:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664795648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239306:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239310:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775737.239312:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239315:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239326:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239330:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775737.239334:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775737.239341:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775737.239345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664795776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239352:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239355:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775737.239357:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239359:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239362:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239364:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775737.239368:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775737.239375:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664795904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775737.239385:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239389:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775737.239394:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239399:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe3480 x1862032664795392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775737.239399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775737.239409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664796032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775737.239422:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775737.239425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775737.239426:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775737.239429:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17504 00000100:00080000:1.0:1775775737.239432:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775737.239433:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664795392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775737.239436:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775737.239441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775737.239451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664796160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775737.239453:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239464:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239470:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664795392:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775737.239480:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775737.239505:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239519:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe1880 x1862032664795520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239532:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664795520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239540:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239543:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239546:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775737.239583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775737.239605:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe2300 x1862032664795648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664795648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239639:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239642:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775737.239648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775737.239668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe2680 x1862032664795776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664795776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239709:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239712:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775737.239720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775737.239740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe2a00 x1862032664795904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239760:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664795904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664795904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775737.239788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775737.239806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239817:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe0380 x1862032664796032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664796032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239837:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775737.239849:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775737.239867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775737.239878:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775737/real 1775775737] req@ffff982981fe0000 x1862032664796160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775737.239890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664796160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775737.239898:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775737.239901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775737.239904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775737.751160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775738.007255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775738.775172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775739.031248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775739.799159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775740.055360:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775740.823201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775741.079296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775741.847158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775742.103310:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775742.359172:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359179:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775742.359183:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359191:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359204:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359209:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775742.359245:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664796288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775742.359272:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359280:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775742.359281:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359284:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359287:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359290:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775742.359298:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775742.359307:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664796416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775742.359314:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359317:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775742.359318:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359321:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359323:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359325:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775742.359329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775742.359331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664796288:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775742.359338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664796544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775742.359345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359348:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775742.359349:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359352:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359354:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359356:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775742.359359:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775742.359366:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664796672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775742.359371:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359374:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775742.359375:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359378:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359388:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359391:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775742.359394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775742.359400:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664796800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775742.359406:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775742.359412:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775742.359415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775742.359416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359420:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359423:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359426:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775742.359430:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775742.359436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664796928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775742.359442:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775742.359445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe0000 x1862032664796288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775742.359446:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775742.359453:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775742.359461:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17509 00000100:00080000:1.0:1775775742.359464:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775742.359467:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359470:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664796288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775742.359472:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775742.359498:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664797056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775742.359504:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359509:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359513:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664796288:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775742.359520:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775742.359540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359549:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe0380 x1862032664796416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359557:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664796416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359566:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775742.359593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775742.359614:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe2a00 x1862032664796544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359636:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664796544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775742.359651:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775742.359666:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359673:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe2680 x1862032664796672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664796672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359694:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775742.359702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775742.359712:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe2300 x1862032664796800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664796800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664796800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775742.359738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775742.359747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe1880 x1862032664796928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664796928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664796928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775742.359772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775742.359783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775742.359789:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775742/real 1775775742] req@ffff982981fe3480 x1862032664797056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775742.359796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664797056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775742.359801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775742.359803:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775742.359805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775742.871183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775743.127267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775743.895145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775744.151250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775744.919174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775745.175248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775745.943232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775746.199257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775746.967177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775747.223254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775747.479250:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775747.479258:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775747.479262:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479271:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479281:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775747.479287:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775747.479320:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664797184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775747.479361:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775747.479364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775747.479365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479368:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479372:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775747.479377:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775747.479424:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664797184:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775747.479425:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664797312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775747.479440:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775747.479444:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775747.479446:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479450:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479454:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775747.479458:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479463:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775747.479475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664797440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775747.479484:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775747.479489:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775747.479491:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775747.479494:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479499:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479505:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775747.479509:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479515:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775747.479522:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b4700 x1862032664797184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775747.479528:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664797568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775747.479541:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775747.479543:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664797184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775747.479550:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775747.479555:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775747.479558:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775747.479562:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:0.0:1775775747.479565:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775747.479569:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775747.479570:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664797184:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775747.479578:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479584:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775747.479594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775747.479600:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664797696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775747.479612:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775747.479616:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775747.479620:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775747.479623:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479627:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479630:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775747.479631:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b5c00 x1862032664797312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775747.479637:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775747.479651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664797312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775775747.479660:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664797824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775747.479664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479672:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775747.479683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797440:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775747.479687:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775747.479690:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775747.479691:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775747.479696:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17514 00000100:00080000:2.0:1775775747.479699:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775747.479702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775747.479706:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775747.479710:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775747.479714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b5180 x1862032664797440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775747.479720:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664797952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775747.479729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664797440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775747.479744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775747.479753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775747.479766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775747.479772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b6d80 x1862032664797568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775747.479780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664797568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775747.479787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775747.479838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775747.479851:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775747.479858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b4a80 x1862032664797696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775747.479866:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664797696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775747.479873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664797696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775747.479880:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775747.479890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775747.479895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b5500 x1862032664797824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775747.479902:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664797824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479907:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775747.479909:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479910:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775747.479914:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775747.479923:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775747.479929:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775747/real 1775775747] req@ffff9829764b4700 x1862032664797952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775747.479936:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664797952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775747.479941:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775747.479943:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775747.479945:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664797952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775747.991218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775748.247234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775749.015196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775749.271266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775750.039212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775750.295203:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775751.063176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775751.319286:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775775752.087180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775752.343265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775752.599257:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599270:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775752.599275:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599286:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599325:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599344:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775752.599391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664798080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775752.599421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599425:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775752.599427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599430:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599434:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599437:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775752.599457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664798208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775752.599465:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599467:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775752.599469:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599472:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599478:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599482:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775752.599488:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664798080:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775752.599496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664798336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775752.599504:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599507:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775752.599507:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599510:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599513:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599516:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599521:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775752.599533:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664798464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775752.599544:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599548:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775752.599550:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599555:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599559:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599563:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599569:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775752.599583:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664798592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775752.599590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775752.599602:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599606:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775752.599608:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599614:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00080000:0.0:1775775752.599619:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599624:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599630:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775752.599643:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664798720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775752.599649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2e680 x1862032664798080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775752.599665:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775752.599669:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775752.599671:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775752.599683:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17520 00000100:00100000:3.0:1775775752.599683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664798080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775752.599690:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775752.599696:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775752.599703:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775752.599707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.599714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775752.599716:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664798848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775752.599723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664798080:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775752.599736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775752.599767:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.599781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2d500 x1862032664798208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.599794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664798208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.599808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.599812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.599815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775752.599847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775752.599869:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.599880:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2f480 x1862032664798336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.599891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664798336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.599899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.599902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.599905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775752.599911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775752.599933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.599944:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2fb80 x1862032664798464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.599955:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664798464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.599963:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.599966:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.599970:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775752.599976:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775752.599997:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.600008:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2e300 x1862032664798592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.600020:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664798592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.600028:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.600031:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.600035:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664798592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775752.600041:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664798720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775752.600107:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.600121:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2d180 x1862032664798720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.600135:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664798720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.600143:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.600146:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.600160:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664798720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775752.600166:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664798848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775752.600187:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775752.600199:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775752/real 1775775752] req@ffff982974b2c380 x1862032664798848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775752.600212:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664798848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775752.600221:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775752.600225:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775752.600229:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664798848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775753.111219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775753.367269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775754.135205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775754.391321:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775755.159216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775755.415301:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775756.183127:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775756.439223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775757.207160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775757.463280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775757.719180:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719191:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775757.719196:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719209:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719234:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719242:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775757.719296:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664798976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775757.719322:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719325:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775757.719327:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719331:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719335:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719339:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775757.719349:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775757.719361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664799104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775757.719361:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664798976:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775757.719392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775757.719396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719402:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719404:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719406:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775757.719410:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775757.719428:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664799232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775757.719438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719441:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775757.719444:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719447:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719450:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719452:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775757.719456:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775757.719457:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775757.719476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664799360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775757.719485:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719489:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775757.719491:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719498:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00100000:3.0:1775775757.719500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2df80 x1862032664798976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775757.719502:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719521:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719529:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664798976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775757.719533:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775757.719553:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.719561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775757.719564:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664799488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775757.719570:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664798976:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775757.719589:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719592:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775757.719594:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719598:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719605:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719610:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775757.719615:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775757.719625:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664799616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775757.719632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799104:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775757.719635:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775757.719638:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775757.719642:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775757.719652:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17525 00000100:00080000:0.0:1775775757.719654:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775757.719656:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775757.719660:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775757.719669:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664799744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775757.719685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719705:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2ca80 x1862032664799104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.719720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664799104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.719729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.719733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.719737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775757.719744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775757.719765:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2f100 x1862032664799232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.719787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664799232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.719795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.719799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.719802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775757.719808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775757.719830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719841:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2c380 x1862032664799360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.719853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664799360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.719862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.719866:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.719870:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775757.719876:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775757.719896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2d180 x1862032664799488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.719921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664799488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.719929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.719932:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.719936:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664799488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775757.719957:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664799616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775757.719978:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.719988:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2e300 x1862032664799616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.720000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664799616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.720009:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.720013:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.720016:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664799616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775757.720022:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664799744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775757.720042:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775757.720053:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775757/real 1775775757] req@ffff982974b2fb80 x1862032664799744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775757.720089:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664799744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775757.720099:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775757.720102:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775757.720105:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664799744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775758.231195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775758.487278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775759.255160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775759.511296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775760.279162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775760.535289:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775761.303176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775761.559377:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775762.327314:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775762.583317:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775762.839141:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839151:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775762.839157:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839169:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839181:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775762.839187:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775762.839217:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664799872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775762.839240:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839247:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775762.839249:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839253:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839256:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775762.839259:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839267:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775762.839274:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664799872:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775762.839279:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664800000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775762.839288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775762.839292:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839295:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775762.839300:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839304:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775762.839313:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664800128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775762.839320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775762.839324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839327:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839330:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775762.839333:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775762.839346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664800256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775762.839352:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775762.839356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839359:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839361:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775762.839364:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839367:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775762.839372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664800384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775762.839380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800000:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775762.839395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775762.839403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775762.839414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800128:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775762.839417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839425:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775762.839440:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775762.839442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664800512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775762.839462:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800384:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775775762.839463:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775762.839469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775762.839471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775762.839476:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17530 00000100:00080000:0.0:1775775762.839481:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775762.839483:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664800512:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775762.839488:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775762.839493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775762.839506:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664800640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775762.839533:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664800640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775763.351184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775763.607269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775764.375227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775764.631263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775765.399140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775765.655191:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775765.911392:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775765.911464:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775765.911487:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911542:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911553:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911574:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911585:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911597:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775765.911607:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775765.911615:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2e300 x1862032664800384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664800384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911689:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911695:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775765.911706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2d500 x1862032664800000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664800000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775765.911744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2e680 x1862032664800128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664800128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775765.911783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2fb80 x1862032664800256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664800256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911807:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911811:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775765.911823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2f480 x1862032664799872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664799872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911848:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664799872:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775765.911887:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2c380 x1862032664800640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664800640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911906:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911910:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911913:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664800640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775765.911920:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775762/real 1775775765] req@ffff982974b2d180 x1862032664800512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775765.911930:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664800512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775765.911938:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775765.911942:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775765.911945:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664800512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775766.423201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775766.679282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775767.447149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775767.703240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775767.959217:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959230:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775767.959236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959246:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959259:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959265:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775767.959299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664800768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775767.959339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775767.959366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664800896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775767.959379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775767.959399:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664801024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959408:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775775767.959414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664800768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775767.959415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959424:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959431:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959438:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775767.959447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664801152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775767.959458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959461:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959465:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959468:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775767.959477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664801280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959486:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959490:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775767.959492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959495:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00080000:2.0:1775775767.959509:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959513:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775767.959526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775767.959550:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664801408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775767.959562:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775767.959566:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775767.959568:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775767.959582:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17535 00000100:00100000:3.0:1775775767.959589:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b5f80 x1862032664800768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775767.959591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775767.959603:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775767.959619:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775767.959638:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664800768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775767.959648:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664801536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775767.959664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.959676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.959684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664800768:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775767.959700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775767.959740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.959756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b6680 x1862032664800896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.959780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664800896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.959791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.959795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.959800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664800896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775767.959838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775767.959864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.959998:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b4e00 x1862032664801024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.960012:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664801024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.960022:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.960026:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.960029:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775767.960051:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775767.960119:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.960132:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b7b80 x1862032664801152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.960145:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664801152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.960153:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.960156:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.960160:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775767.960167:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775767.960188:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.960199:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b7100 x1862032664801280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.960210:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664801280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.960218:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.960222:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.960225:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775767.960232:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664801408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775767.960258:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.960270:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b6a00 x1862032664801408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.960283:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664801408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.960293:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.960297:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.960301:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664801408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775767.960308:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664801536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775767.960332:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775767.960343:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775767/real 1775775767] req@ffff9829764b4000 x1862032664801536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775767.960355:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664801536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775767.960364:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775767.960368:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775767.960371:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664801536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775768.471164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775768.727249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775769.495227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775769.751301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775770.519230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775770.775255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775771.543181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775771.799238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775772.567169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775772.823297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775773.079204:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079214:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775773.079219:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079230:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079245:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775773.079298:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664801664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775773.079331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775773.079335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079340:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079342:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079345:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775773.079354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775773.079364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664801792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775773.079378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775773.079382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079388:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00100000:3.0:1775775773.079388:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664801664:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775773.079394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775773.079405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775773.079421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664801920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775773.079433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775773.079439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079443:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079457:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079462:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775773.079467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775773.079470:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775773.079480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664802048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775773.079489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775773.079495:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079498:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079502:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079505:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775773.079509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775773.079516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664802176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775773.079525:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775775773.079529:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b4000 x1862032664801664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775773.079531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775773.079541:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079547:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079552:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079555:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664801664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775773.079559:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775773.079570:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079575:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775773.079580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664802304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775773.079580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664801664:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775773.079603:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775773.079606:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775773.079608:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775773.079609:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801792:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775773.079614:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17540 00000100:00080000:2.0:1775775773.079619:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775773.079622:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775773.079627:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775773.079640:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664802432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775773.079645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079660:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b6a00 x1862032664801792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079670:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664801792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.079677:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079680:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.079683:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775773.079689:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775773.079704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b7100 x1862032664801920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079718:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664801920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.079724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.079727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664801920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775773.079731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775773.079743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b7b80 x1862032664802048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664802048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.079761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.079764:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775773.079768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775773.079779:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b4e00 x1862032664802176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664802176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.079797:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.079801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775773.079804:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664802304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775773.079820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b6680 x1862032664802304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664802304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.079840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.079843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.079845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664802304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775773.079926:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664802432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775773.079966:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775773.079975:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775773/real 1775775773] req@ffff9829764b5f80 x1862032664802432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775773.079988:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664802432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775773.080005:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775773.080007:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775773.080009:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664802432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775773.591243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775773.847305:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775774.615196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775774.871293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775775.639214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775775.895285:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775776.663188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775776.919254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775777.687201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775777.943264:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775778.199184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199192:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775778.199197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199242:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199249:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775778.199303:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664802560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199329:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199332:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775778.199334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199338:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199344:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775778.199385:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664802688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199398:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775778.199400:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199405:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199408:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199410:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775778.199427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664802816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775775778.199447:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664802560:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775778.199448:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199469:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199472:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775778.199491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664802944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199499:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775778.199504:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199508:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199510:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199513:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199517:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775778.199527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664803072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775778.199540:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199544:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199546:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199549:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199554:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775775778.199562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775778.199573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664803200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775778.199583:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775778.199586:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775778.199588:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775778.199596:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17545 00000100:00080000:0.0:1775775778.199599:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775778.199605:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775778.199610:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775778.199619:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664803328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775778.199634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2f100 x1862032664802560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664802560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199676:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199681:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664802560:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775778.199695:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775778.199718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.199737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2ca80 x1862032664802688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664802688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775778.199790:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775778.199803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.199810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2df80 x1862032664802816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664802816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775778.199833:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775778.199846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.199852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2d180 x1862032664802944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664802944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199888:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199893:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664802944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775778.199897:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775778.199914:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.199921:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2c380 x1862032664803072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664803072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199936:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199938:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775778.199942:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664803200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775778.199954:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.199959:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2f480 x1862032664803200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.199967:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664803200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.199972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.199980:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.199983:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664803200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775778.199988:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664803328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775778.200000:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775778.200007:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775778/real 1775775778] req@ffff982974b2fb80 x1862032664803328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775778.200014:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664803328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775778.200019:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775778.200022:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775778.200024:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664803328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775778.711164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775778.967253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775779.735200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775779.991308:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775780.759161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775781.015277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775781.783225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775782.039270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775782.807159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775783.064533:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775783.319225:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319236:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775783.319241:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319252:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319266:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319273:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775783.319312:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664803456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775783.319342:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319346:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775783.319348:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319353:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319357:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319362:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775783.319373:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775783.319387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664803584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775783.319399:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319403:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775783.319405:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319414:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319418:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775783.319425:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775783.319441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664803712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775783.319452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664803456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775783.319454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775783.319472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319477:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775783.319491:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775783.319507:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664803840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775783.319518:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775783.319524:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319530:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319557:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319563:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775783.319568:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775783.319575:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775783.319588:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664803968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775783.319601:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775783.319608:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775783.319612:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319618:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319623:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775783.319628:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775783.319635:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b4700 x1862032664803456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775783.319649:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664804096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775783.319665:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775783.319668:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775783.319672:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775783.319674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664803456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775783.319688:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17550 00000100:00080000:2.0:1775775783.319700:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775783.319706:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775783.319707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775775783.319716:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775783.319719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775783.319733:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664804224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775783.319796:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664803456:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775783.319809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775783.319841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.319857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b5500 x1862032664803584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.319872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664803584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.319882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.319886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.319889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775783.319923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775783.319947:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.319959:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b4a80 x1862032664803712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.319984:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664803712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.319994:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.319998:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.320002:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775783.320009:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775783.320032:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.320045:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b6d80 x1862032664803840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.320058:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664803840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.320111:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.320116:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.320119:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775783.320126:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775783.320152:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.320164:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b5180 x1862032664803968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.320186:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664803968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.320196:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.320200:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.320203:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664803968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775783.320210:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775783.320233:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.320252:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b5c00 x1862032664804096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.320266:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664804096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.320275:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.320279:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.320283:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775783.320289:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775783.320312:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775783.320324:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775783/real 1775775783] req@ffff9829764b5f80 x1862032664804224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775783.320337:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664804224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775783.320346:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775783.320349:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775783.320352:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775783.831200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775784.087240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775784.855208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775785.111262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775785.879187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775786.135265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775786.903174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775787.159245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775787.927225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775788.183271:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775788.439198:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439207:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775788.439211:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439294:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439303:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775788.439343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664804352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775788.439374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439381:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439384:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775788.439393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775788.439401:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664804480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775788.439413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439415:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439417:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439421:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439424:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664804352:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775788.439425:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775788.439435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664804608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775788.439446:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439449:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439451:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439453:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775788.439456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775788.439465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664804736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775788.439479:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439482:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439487:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439489:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775788.439492:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775788.439498:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664804864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439504:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775788.439507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439509:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439510:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439512:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775788.439516:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775788.439521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664804992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775788.439527:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775788.439528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775788.439529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775788.439531:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17555 00000100:00080000:3.0:1775775788.439533:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775788.439535:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775788.439537:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775788.439542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664805120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775788.439590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9df80 x1862032664804352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664804352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664804352:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775788.439716:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775788.439759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9c000 x1862032664804480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664804480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439803:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775788.439831:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775788.439845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9d180 x1862032664804608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664804608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439865:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439869:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775788.439873:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775788.439885:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9f480 x1862032664804736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439898:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664804736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439908:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439913:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775788.439916:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775788.439927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439932:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9ea00 x1862032664804864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439939:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664804864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439945:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439946:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664804864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775788.439952:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775788.439962:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.439968:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9dc00 x1862032664804992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.439975:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664804992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.439980:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.439982:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.439984:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664804992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775788.439988:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664805120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775788.440000:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775788.440006:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775788/real 1775775788] req@ffff982980f9e300 x1862032664805120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775788.440012:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664805120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775788.440017:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775788.440018:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775788.440020:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664805120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775788.951118:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775789.207281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775789.975201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775790.231223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775790.999206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775791.255248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775792.023227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775792.279253:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775793.047200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775793.303290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775793.559160:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559170:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775793.559174:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559182:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559193:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559199:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775793.559239:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664805248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559275:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559282:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775793.559290:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559297:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559303:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559307:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559314:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664805248:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775793.559316:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775793.559332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664805376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559340:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559343:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775793.559345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559348:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559350:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559353:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775793.559356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775793.559364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664805504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775793.559374:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559377:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17561 00000800:00000100:0.0:1775775793.559382:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775793.559384:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559388:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775793.559392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775793.559401:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664805632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775793.559415:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b6680 x1862032664805248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775793.559415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559419:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559424:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559430:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775793.559436:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775793.559443:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664805248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775793.559444:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664805760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559470:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775793.559472:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775793.559475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775793.559478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775793.559479:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559485:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775793.559488:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775793.559492:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664805248:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775793.559497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775793.559510:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664805888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775793.559522:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775793.559525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805376:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775793.559526:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775793.559530:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775793.559535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17561 00000100:00080000:2.0:1775775793.559540:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775793.559546:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775793.559550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775793.559556:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775793.559563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b4e00 x1862032664805376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775793.559573:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664806016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775793.559576:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664805376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559604:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559610:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775793.559616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775793.559638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559647:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b7b80 x1862032664805504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775793.559656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664805504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775793.559670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775793.559681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b7100 x1862032664805632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775793.559695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664805632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775793.559708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775793.559718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b6a00 x1862032664805760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775793.559731:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664805760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664805760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775793.559752:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664805888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775793.559762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b4000 x1862032664805888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775793.559776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664805888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664805888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775793.559788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775793.559797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775793.559802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775793/real 1775775793] req@ffff9829764b6680 x1862032664806016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775793.559809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664806016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775793.559815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775793.559816:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775793.559818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775794.071184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775794.327310:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775795.095213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775795.351267:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775796.119199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775796.375340:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775797.143184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775797.399196:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775798.167178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775798.423296:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775798.679790:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.679805:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775798.679818:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.679834:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.679851:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.679860:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775798.679912:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664806144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775798.679956:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.679959:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775798.679961:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.679966:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.679970:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.679974:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775798.679985:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775798.680001:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664806272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775798.680012:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.680020:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775798.680025:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664806144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775798.680025:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.680034:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.680045:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.680050:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775798.680054:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775798.680142:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664806400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775798.680157:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775798.680189:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.680194:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775798.680198:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.680206:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.680213:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.680219:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680226:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9f100 x1862032664806144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775798.680229:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775798.680263:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664806528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775798.680277:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664806144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775798.680278:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.680286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775798.680288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.680300:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.680311:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775798.680322:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775798.680323:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775798.680335:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775775798.680336:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775798.680346:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664806144:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775798.680357:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664806656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775798.680367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.680371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775798.680373:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775798.680378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00100000:0.0:1775775798.680384:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806272:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775798.680386:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.680397:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775798.680404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775798.680424:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664806784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775798.680425:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680443:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9d880 x1862032664806272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775798.680447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775798.680459:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775798.680462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775798.680464:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664806272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775798.680475:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17566 00000100:00080000:3.0:1775775798.680481:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775798.680487:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775798.680493:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775798.680495:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775798.680519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680525:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775798.680533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664806912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775798.680540:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775798.680581:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9ce00 x1862032664806400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775798.680600:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664806400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775798.680606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775798.680608:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775798.680615:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775798.680635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680647:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9e300 x1862032664806528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775798.680659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664806528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775798.680668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775798.680672:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775798.680683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775798.680702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9dc00 x1862032664806656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775798.680718:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664806656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775798.680723:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775798.680725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664806656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775798.680746:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775798.680766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9ea00 x1862032664806784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775798.680791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664806784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775798.680799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775798.680802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775798.680808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775798.680824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775798.680831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775798/real 1775775798] req@ffff982980f9f480 x1862032664806912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775798.680839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664806912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775798.680844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775798.680846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775798.680848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664806912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775799.191198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775799.447278:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775800.215235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775800.471263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775801.239206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775801.495283:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775802.263177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775802.519262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775803.287199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775803.543249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775803.799250:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799261:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775803.799266:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799277:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799290:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799297:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775803.799337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664807040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799376:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775803.799378:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799383:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799387:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799392:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775803.799434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664807168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799452:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775803.799454:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775803.799465:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664807040:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775803.799465:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799472:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799483:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775803.799507:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664807296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799517:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775803.799524:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799528:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799533:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799536:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799542:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775803.799555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664807424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799564:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775803.799569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799573:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799577:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799581:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799585:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775803.799596:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664807552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799605:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799608:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775803.799610:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799613:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799617:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799620:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799624:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775803.799632:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664807680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775803.799641:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775803.799644:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775803.799646:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775803.799649:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17571 00000100:00080000:2.0:1775775803.799652:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775803.799656:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775803.799660:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775803.799671:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664807808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775803.800877:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.800924:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b5f80 x1862032664807040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.800947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664807040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.800960:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.800966:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.800971:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664807040:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775803.801004:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775803.801031:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801044:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b5c00 x1862032664807168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801055:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664807168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801081:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801085:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801089:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775803.801095:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775803.801117:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801129:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b5180 x1862032664807296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801143:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664807296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801151:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801154:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801158:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775803.801164:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775803.801185:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801195:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b6d80 x1862032664807424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801206:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664807424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801213:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801217:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801220:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775803.801226:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775803.801244:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801253:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b4a80 x1862032664807552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801265:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664807552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801273:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801277:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801280:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664807552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775803.801286:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664807680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775803.801305:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801316:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b5500 x1862032664807680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801334:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664807680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801342:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801345:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801348:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664807680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775803.801354:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664807808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775803.801377:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775803.801388:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775803/real 1775775803] req@ffff9829764b4700 x1862032664807808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775803.801397:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664807808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775803.801403:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775803.801405:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775803.801422:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664807808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775804.311195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775804.567218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775805.335205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775805.591290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775806.359211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775806.615263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775807.383217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775807.639305:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775808.407314:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775808.663310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775808.919169:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.919178:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775808.919181:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920249:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920260:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920281:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775808.920317:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664807936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920352:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775808.920357:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920361:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920368:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920377:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775808.920387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664808064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775808.920401:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664807936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775808.920402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920412:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920415:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775808.920427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664808192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775808.920437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920439:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920441:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920444:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775808.920453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664808320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920461:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775808.920463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775808.920468:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775808.920470:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920474:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920476:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775808.920488:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664808448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775808.920499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920501:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920503:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775808.920506:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920509:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775808.920512:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b4700 x1862032664807936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775808.920519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664808576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775808.920546:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775808.920550:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775808.920551:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775808.920554:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17576 00000100:00080000:2.0:1775775808.920558:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775808.920559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664807936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775808.920563:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775808.920573:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775808.920575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920585:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664807936:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775808.920590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664808704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775808.920608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775808.920627:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b5500 x1862032664808064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664808064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920654:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775808.920658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775808.920668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b4a80 x1862032664808192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664808192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920694:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920696:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775808.920699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775808.920710:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b6d80 x1862032664808320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664808320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920732:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775808.920736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775808.920745:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b5180 x1862032664808448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664808448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775808.920771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664808576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775808.920780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b5c00 x1862032664808576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664808576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920797:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664808576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775808.920804:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664808704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775808.920813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775808.920819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775808/real 1775775808] req@ffff9829764b5f80 x1862032664808704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775808.920826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664808704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775808.920830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775808.920832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775808.920834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664808704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775809.431194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775809.687273:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775810.455171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775810.711260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775811.479175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775811.735260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775812.503130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775812.759235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775813.527186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775813.783303:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775814.039171:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039178:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775814.039183:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775814.039191:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039208:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039214:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775814.039267:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664808832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039295:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039299:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775814.039301:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775814.039305:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039309:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039314:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039324:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775814.039339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664808960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039349:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775814.039353:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775814.039358:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664808832:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775814.039358:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039361:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039367:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039372:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775814.039381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664809088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039387:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775814.039392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775814.039395:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039398:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039400:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039404:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775814.039411:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664809216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039420:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775814.039421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775814.039423:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039426:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039430:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775775814.039433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039434:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775814.039444:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664809344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775814.039455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775814.039457:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039460:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775814.039463:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039467:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775814.039474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b6680 x1862032664808832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775814.039476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664809472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775814.039511:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775814.039515:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775814.039518:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775814.039519:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664808832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775814.039527:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17581 00000100:00080000:2.0:1775775814.039535:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775814.039541:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775814.039541:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775814.039550:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775814.039551:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039557:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664808832:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775814.039564:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664809600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775814.039584:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775814.039608:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039620:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b4000 x1862032664808960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664808960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039649:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664808960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775814.039655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775814.039672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039681:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b6a00 x1862032664809088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664809088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039708:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775814.039714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775814.039731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b7100 x1862032664809216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039753:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664809216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039761:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775814.039773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775814.039789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039797:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b7b80 x1862032664809344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664809344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775814.039837:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664809472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775814.039852:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b4e00 x1862032664809472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664809472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039880:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039887:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664809472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775814.039892:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664809600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775814.039907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775814.039915:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775814/real 1775775814] req@ffff9829764b5f80 x1862032664809600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775814.039927:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664809600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775814.039935:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775814.039938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775814.039942:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664809600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775814.551177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775814.807319:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775815.575148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775815.831250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775816.599253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775816.855269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775817.623179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775817.879236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775818.647200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775818.903250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775819.159280:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159295:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775819.159301:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775819.159315:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159331:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159340:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775819.159393:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664809728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775819.159435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775819.159440:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159445:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159450:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775819.159464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775819.159483:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664809856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775819.159502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775819.159507:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664809728:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775819.159510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159517:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159522:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775819.159528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775819.159541:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664809984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159552:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159556:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775819.159558:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775819.159563:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159567:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159572:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775819.159579:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775819.159593:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664810112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159604:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775819.159611:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775819.159616:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159620:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159624:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775819.159630:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775819.159634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775819.159646:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664810240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159658:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159662:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775819.159664:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775819.159668:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159673:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159678:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775819.159684:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775819.159695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9d180 x1862032664809728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775819.159699:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664810368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775819.159725:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775819.159729:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775819.159732:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775819.159734:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664809728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775819.159740:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17586 00000100:00080000:3.0:1775775819.159748:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775819.159754:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775819.159755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775819.159762:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775819.159763:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.159770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664809728:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775819.159778:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664810496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775819.159802:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775819.159830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.159844:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9c000 x1862032664809856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.159858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664809856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.159867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.159870:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.159873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775819.159881:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775819.159903:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.159914:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9df80 x1862032664809984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.159926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664809984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.159934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.159938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.159941:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664809984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775819.159947:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775819.159967:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.159977:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9c380 x1862032664810112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.159987:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664810112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.159996:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.159999:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.160002:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775819.160007:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775819.160026:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.160048:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9f800 x1862032664810240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.160061:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664810240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.160104:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.160107:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.160112:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775819.160118:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664810368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775819.160156:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.160169:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9c700 x1862032664810368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.160182:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664810368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.160190:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.160193:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.160196:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664810368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775819.160202:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664810496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775819.160224:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775819.160235:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775819/real 1775775819] req@ffff982980f9e680 x1862032664810496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775819.160247:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664810496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775819.160256:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775819.160259:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775819.160262:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664810496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775819.671185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775819.927280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775820.695191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775820.951281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775821.719198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775821.976540:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775822.743183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775822.999245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775823.767201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775824.023190:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775824.279190:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279198:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775824.279202:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279211:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279229:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279235:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775824.279267:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664810624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279290:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775824.279294:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279297:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279300:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279311:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775824.279326:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664810624:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775824.279340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664810752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279349:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279352:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775824.279353:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279357:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279359:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279362:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775824.279373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664810880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775824.279383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279385:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279387:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279389:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775824.279398:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664811008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775775824.279414:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775824.279423:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775824.279425:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279436:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775824.279445:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2e680 x1862032664810624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775824.279451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664811136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279469:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775824.279478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775824.279483:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279487:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775824.279490:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279493:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775824.279502:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664810624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775824.279503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664811264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775824.279523:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775824.279529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775824.279530:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775824.279532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775824.279534:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17591 00000100:00080000:0.0:1775775824.279541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775824.279544:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775824.279545:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775824.279551:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775824.279554:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664810624:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775824.279561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664811392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775824.279600:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775824.279622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279631:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2d500 x1862032664810752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279640:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664810752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279647:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775824.279655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775824.279666:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2e300 x1862032664810880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664810880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664810880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775824.279697:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775824.279708:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2fb80 x1862032664811008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664811008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279727:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279729:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775824.279732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775824.279741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2f480 x1862032664811136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664811136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775824.279770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664811264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775824.279780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2c380 x1862032664811264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664811264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664811264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775824.279829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664811392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775824.279849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775824.279868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775824/real 1775775824] req@ffff982974b2d180 x1862032664811392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775824.279876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664811392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775824.279882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775824.279884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775824.279886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664811392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775824.791175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775825.047261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775825.815186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775826.071260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775826.839186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775827.095277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775827.863193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775828.120504:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775828.887240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775829.143298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775829.399191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399249:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775829.399267:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399282:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399333:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399343:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775829.399422:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664811520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399464:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775829.399466:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399478:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399482:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775829.399487:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664811520:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775829.399490:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399503:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775829.399516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664811648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775829.399529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399532:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399534:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399537:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399540:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775829.399547:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664811776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399553:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399556:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775829.399557:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399560:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399561:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399564:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399567:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775829.399572:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664811904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399578:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399580:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775829.399582:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399585:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399587:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399589:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399591:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775829.399599:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664812032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399604:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399606:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775829.399607:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399611:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399613:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399616:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399618:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775829.399626:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664812160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775829.399631:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775829.399633:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775829.399634:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775829.399637:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17596 00000100:00080000:2.0:1775775829.399639:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775829.399642:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775829.399645:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775829.399653:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664812288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775829.401426:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775829.401526:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775829.401573:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775829.401628:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775829.401653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664812160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775829.401674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664812288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775829.911174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775830.167317:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775830.935194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775831.191288:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775831.959178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775832.215250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775775832.471469:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775775832.471547:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775775832.471571:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471628:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471639:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471649:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471660:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471671:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775775832.471684:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775832.471735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b5180 x1862032664811648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.471776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664811648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.471789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.471800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.471806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775832.471815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b6d80 x1862032664811776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.471830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664811776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.471839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.471844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.471848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775832.471856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b4a80 x1862032664811904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.471868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664811904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.471876:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.471880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.471883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664811904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775832.471891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b5500 x1862032664812032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.471905:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664812032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.471944:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.471948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.471951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775832.471960:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b4700 x1862032664812160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.471972:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664812160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.471980:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.471983:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.471986:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664812160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775832.471993:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b5f80 x1862032664812288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.472006:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664812288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.472013:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.472016:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.472019:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664812288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775832.472026:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775829/real 1775775832] req@ffff9829764b5c00 x1862032664811520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775832.472039:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664811520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775832.472046:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775832.472050:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775832.472054:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664811520:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775832.983181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775833.239272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775834.007209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775834.263285:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775834.519237:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519255:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775834.519260:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519274:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519287:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775834.519293:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775834.519336:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664812416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519380:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775834.519386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519397:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775834.519402:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519414:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775834.519430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664812544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519441:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519445:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775834.519447:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519453:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519458:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775834.519462:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775834.519484:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664812672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519501:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519505:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775834.519507:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519513:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519519:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775834.519525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664812416:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775834.519526:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775834.519556:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664812800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519564:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519568:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775834.519570:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519575:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519579:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775834.519584:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519589:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775834.519604:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664812928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519622:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519625:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775834.519627:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519632:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519639:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775834.519640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775834.519647:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519654:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775834.519667:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664813056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775834.519678:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775834.519681:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775834.519683:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775834.519687:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17602 00000100:00080000:3.0:1775775834.519691:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775834.519695:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775834.519701:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775834.519718:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664813184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775834.519747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9d500 x1862032664812416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.519777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664812416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.519789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.519795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.519799:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664812416:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775834.519808:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775834.519839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.519852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9fb80 x1862032664812544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.519863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664812544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.519871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.519908:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.519912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775834.519944:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775834.519969:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.519981:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9ed80 x1862032664812672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.519993:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664812672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.520001:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.520004:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.520007:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775834.520013:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775834.520034:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.520045:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9e680 x1862032664812800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.520058:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664812800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.520126:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.520130:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.520134:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775834.520141:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775834.520165:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.520177:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9c700 x1862032664812928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.520191:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664812928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.520200:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.520204:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.520207:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664812928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775834.520213:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775834.520235:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.520250:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9f800 x1862032664813056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.520263:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664813056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.520273:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.520278:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.520280:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775834.520287:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775834.520308:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775834.520320:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775834/real 1775775834] req@ffff982980f9c380 x1862032664813184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775834.520332:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664813184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775834.520341:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775834.520345:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775834.520349:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775835.031180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775835.287273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775836.055208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775836.311302:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775837.079148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775837.335287:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775838.103207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775838.359285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775839.127226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775839.383235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775839.639196:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639211:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775839.639217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639245:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639253:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775839.639297:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664813312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775839.639324:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639327:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775839.639328:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639335:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639342:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639376:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664813312:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775775839.639396:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775839.639420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664813440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775839.639434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639437:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775839.639438:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639442:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639446:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639449:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775839.639454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775839.639465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664813568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775839.639469:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775839.639479:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639481:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775839.639482:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639486:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639489:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639492:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775839.639496:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775839.639503:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664813696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775839.639510:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639523:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775839.639524:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b4e00 x1862032664813312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775839.639525:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639533:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639537:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639543:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775839.639551:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775839.639560:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664813824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775839.639567:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639570:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775839.639571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775839.639577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664813312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775839.639577:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639582:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639589:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775839.639595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775839.639597:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775839.639605:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664813952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775839.639613:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664813312:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775839.639618:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775839.639624:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775839.639626:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775839.639633:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17607 00000100:00080000:2.0:1775775839.639637:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775839.639641:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775839.639646:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775839.639651:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775839.639663:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664814080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775839.639677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639692:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b7b80 x1862032664813440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664813440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639716:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775839.639720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775839.639734:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b7100 x1862032664813568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639748:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664813568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639756:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775839.639761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775839.639773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b6a00 x1862032664813696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639804:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664813696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775839.639819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775839.639833:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b4000 x1862032664813824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664813824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639860:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664813824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775839.639866:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775839.639876:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b6680 x1862032664813952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664813952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639895:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639896:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639898:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664813952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775839.639911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775839.639922:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775839.639927:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775839/real 1775775839] req@ffff9829764b4e00 x1862032664814080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775839.639934:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664814080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775839.639948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775839.639952:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775839.639955:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775840.151212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775840.407297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775841.175262:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775841.431208:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775842.199182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775842.455241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775843.223169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775843.479235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775844.247175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775844.503294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775844.759240:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775844.759254:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775844.759259:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.759272:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.759288:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.759297:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775844.759351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664814208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775844.759393:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775844.759398:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775844.759400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.759405:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.759409:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.759414:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775844.759426:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775844.759441:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664814336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775844.759447:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664814208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775844.759474:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775844.759482:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775844.759484:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.759490:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.759495:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.759501:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775844.759507:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775844.759524:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664814464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775844.759535:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775844.759540:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775844.759544:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775844.759547:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.759552:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.759558:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.759564:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775844.759571:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775844.759586:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664814592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775844.759591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b5c00 x1862032664814208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775844.759609:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775844.759614:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775844.759617:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.759623:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00100000:0.0:1775775844.759625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664814208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775844.759629:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.759638:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775844.759641:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775775844.759644:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775844.759647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.759653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664814208:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775844.759663:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664814720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775844.759673:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775844.759678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775844.759721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.759732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b5f80 x1862032664814336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.759741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664814336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.759747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.759750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.759752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775844.759757:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775844.759770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.759776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b4700 x1862032664814464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.759784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664814464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.759789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.759791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.759793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775844.759797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775844.759808:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.759814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b5500 x1862032664814592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.759821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664814592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.759827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.759829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.759831:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775844.759834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775844.759845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.759850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b4a80 x1862032664814720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.759858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664814720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.759863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.759865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.759868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664814720:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775775844.761116:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775844.761121:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.761130:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.761140:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.761146:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775844.761157:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775844.761190:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664814848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775844.761213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775844.761216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775844.761217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775844.761220:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17612 00000100:00080000:2.0:1775775844.761222:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775844.761225:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775844.761227:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775844.761236:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664814976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775844.761257:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775844.761315:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.761339:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b6d80 x1862032664814848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.761352:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664814848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.761361:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.761364:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.761367:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775844.761372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775844.761385:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775844.761391:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775844/real 1775775844] req@ffff9829764b5180 x1862032664814976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775844.761403:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664814976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775844.761408:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775844.761410:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775844.761412:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664814976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775845.271154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775845.527250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775846.295151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775846.551276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775847.319188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775847.575259:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775848.343277:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775848.599268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775849.367180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775849.623252:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775849.879211:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879218:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775849.879222:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879230:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879241:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879246:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775849.879281:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664815104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775849.879302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879305:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775849.879306:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879309:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879311:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879314:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879320:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775849.879329:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664815232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775849.879337:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775849.879340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879345:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775849.879361:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664815360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775849.879364:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664815104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775849.879369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775849.879376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879379:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879381:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879383:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879386:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775849.879396:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664815488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775849.879403:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879405:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775849.879406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879409:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879411:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879414:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775849.879423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664815616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775849.879429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775849.879432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879438:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879440:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775849.879451:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775849.879452:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664815744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775849.879460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775849.879462:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775849.879464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775849.879467:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17617 00000100:00080000:1.0:1775775849.879470:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775849.879472:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775849.879476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775849.879482:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664815872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775849.879494:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe3b80 x1862032664815104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879515:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664815104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879529:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879533:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664815104:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775849.879541:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775849.879566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879580:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe1f80 x1862032664815232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664815232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775849.879674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775849.879695:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe0a80 x1862032664815360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664815360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775849.879740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775849.879758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe0e00 x1862032664815488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879782:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664815488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879796:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879810:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775849.879815:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775849.879832:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe1c00 x1862032664815616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664815616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879859:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664815616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775849.879864:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664815744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775849.879877:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879885:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe1500 x1862032664815744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664815744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879898:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879900:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664815744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775849.879906:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664815872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775849.879918:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775849.879926:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775849/real 1775775849] req@ffff982981fe1180 x1862032664815872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775849.879933:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664815872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775849.879939:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775849.879941:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775849.879943:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664815872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775850.391200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775850.647261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775851.415196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775851.671258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775852.439194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775852.695285:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775853.463202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775853.719265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775854.487163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775854.743311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775854.999246:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999254:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775854.999258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999275:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999287:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999292:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775854.999332:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664816000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775854.999361:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999364:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999366:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999369:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999376:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775854.999385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664816128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775854.999396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999399:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999401:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999403:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775854.999413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664816256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999423:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775854.999425:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775854.999433:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664816000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775854.999441:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999447:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999450:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775854.999463:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664816384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999469:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999472:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775854.999473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999475:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999478:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999481:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999485:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775854.999494:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664816512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999499:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999501:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775854.999504:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999507:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999511:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999513:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999540:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775854.999541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775854.999557:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664816640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775854.999566:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775854.999569:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775854.999571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775854.999573:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17622 00000100:00080000:2.0:1775775854.999579:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775854.999582:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775854.999586:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775854.999595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b5180 x1862032664816000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775854.999599:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664816768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775854.999669:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664816000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999681:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664816000:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775854.999699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775854.999720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b6d80 x1862032664816128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664816128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775854.999773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775854.999785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b4a80 x1862032664816256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664816256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999805:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999809:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775854.999813:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775854.999824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999832:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b5500 x1862032664816384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664816384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775854.999852:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775854.999863:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999869:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b4700 x1862032664816512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664816512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999881:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999883:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664816512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775854.999889:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664816640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775854.999899:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b5f80 x1862032664816640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664816640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664816640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775854.999925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664816768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775854.999935:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775854.999941:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775854/real 1775775854] req@ffff9829764b5c00 x1862032664816768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775854.999947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664816768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775854.999952:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775854.999955:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775854.999957:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664816768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775855.511218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775855.767327:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775856.535215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775856.791314:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775857.559282:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775857.815336:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775858.583133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775858.839213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775859.607184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775859.863285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775860.119259:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119275:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775860.119280:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119294:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119311:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775860.119321:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775860.119372:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664816896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775860.119404:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119408:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775860.119410:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119413:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119417:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775860.119421:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119443:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775860.119458:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664817024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775860.119467:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664816896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775860.119469:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119476:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775860.119478:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119484:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119488:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775860.119492:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775860.119513:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664817152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775860.119525:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119529:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775860.119532:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119536:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119541:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775775860.119544:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775860.119547:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119553:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775860.119565:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664817280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775860.119572:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775860.119578:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119582:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119584:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775860.119589:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9df80 x1862032664816896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775860.119597:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119603:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775860.119612:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664817408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775860.119620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664816896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775860.119622:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119628:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775860.119631:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119639:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119645:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775860.119652:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119659:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775860.119662:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775775860.119666:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664817536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775860.119668:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664816896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775860.119684:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775860.119687:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775860.119688:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775860.119692:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17627 00000100:00080000:3.0:1775775860.119694:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775860.119697:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775860.119700:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775860.119707:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664817664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775860.119715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775860.119741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9c000 x1862032664817024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119762:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664817024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.119768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.119770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775860.119777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775860.119789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119795:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9d180 x1862032664817152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664817152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.119814:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.119818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775860.119828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775860.119850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119862:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9f480 x1862032664817280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664817280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.119878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.119880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775860.119887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775860.119901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119907:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9ea00 x1862032664817408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664817408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.119920:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.119921:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119923:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775860.119927:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664817536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775860.119937:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119944:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9dc00 x1862032664817536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664817536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.119956:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.119958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.119959:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664817536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775860.119975:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664817664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775860.119986:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775860.119992:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775860/real 1775775860] req@ffff982980f9e300 x1862032664817664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775860.119999:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664817664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775860.120004:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775860.120006:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775860.120008:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664817664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775860.631280:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775860.887229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775861.655171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775861.911221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775862.679203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775862.935244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775863.703186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775863.959277:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775864.727180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775864.983256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775865.239148:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239157:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775865.239161:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239171:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239182:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239187:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775865.239224:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664817792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775865.239256:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239259:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775865.239261:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239265:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239268:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239272:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239281:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775865.239290:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664817920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775865.239298:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239301:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775865.239303:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239306:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239309:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239311:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239316:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775865.239323:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664818048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775865.239330:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239333:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775865.239335:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239351:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239356:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239360:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239367:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775865.239379:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664818176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775865.239390:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664817792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775865.239390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239400:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775865.239402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239407:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239410:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239413:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239418:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775865.239428:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664818304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775865.239436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775865.239439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775865.239441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239444:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239447:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239450:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775865.239462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664818432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775865.239476:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775865.239479:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775865.239480:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775865.239483:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775865.239488:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17632 00000100:00080000:3.0:1775775865.239492:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775865.239495:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775865.239500:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775865.239508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664818560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775865.239533:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9ce00 x1862032664817792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239561:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664817792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239572:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239577:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239581:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664817792:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775865.239590:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775865.239614:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.239628:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9d880 x1862032664817920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664817920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239661:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664817920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775865.239675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775865.239696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.239708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9f100 x1862032664818048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664818048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775865.239774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775865.239797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.239808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9e300 x1862032664818176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664818176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239836:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775865.239842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775865.239861:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.239872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9dc00 x1862032664818304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239885:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664818304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239893:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239901:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775865.239907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664818432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775865.239927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.239938:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9ea00 x1862032664818432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.239951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664818432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.239959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.239963:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.239966:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664818432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775865.239972:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664818560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775865.239993:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775865.240004:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775865/real 1775775865] req@ffff982980f9f480 x1862032664818560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775865.240016:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664818560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775865.240025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775865.240029:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775865.240032:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664818560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775865.751159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775866.007281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775866.775192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775867.031296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775867.799172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775868.055190:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775868.823181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775869.079318:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775869.847156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775870.103268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775870.359197:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359208:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775870.359213:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359225:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359239:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359245:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775870.359298:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664818688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775870.359320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359323:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775870.359325:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775870.359349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664818816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775870.359357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359359:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775870.359360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359363:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359365:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359368:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359371:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775870.359375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664818688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775870.359381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664818944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775870.359392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775870.359396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359399:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359401:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359403:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359406:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775870.359415:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664819072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775870.359421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359424:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775870.359425:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359427:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359429:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359432:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775870.359441:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664819200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775870.359447:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775870.359453:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359455:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775870.359457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359460:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359464:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359467:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359470:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775870.359477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664819328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775870.359483:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775870.359487:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775870.359488:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe1180 x1862032664818688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775870.359488:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775870.359495:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17637 00000100:00080000:1.0:1775775870.359499:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775870.359505:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775870.359511:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775870.359519:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664819456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775870.359524:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664818688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359537:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359542:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359547:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664818688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775870.359555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775870.359577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe1500 x1862032664818816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359600:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664818816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359611:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359614:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775870.359636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775870.359653:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359663:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe1c00 x1862032664818944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664818944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359692:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359695:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664818944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775870.359701:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775870.359737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe0e00 x1862032664819072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664819072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775870.359778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775870.359795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe0a80 x1862032664819200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664819200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359873:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775870.359879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664819328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775870.359897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359907:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe1f80 x1862032664819328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664819328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359927:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359933:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664819328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775870.359938:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664819456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775870.359956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775870.359965:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775870/real 1775775870] req@ffff982981fe3b80 x1862032664819456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775870.359976:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664819456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775870.359984:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775870.359987:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775870.359991:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664819456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775870.871194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775871.127301:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775871.895157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775872.151255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775872.919165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775873.175317:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775775873.943148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775874.199200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775874.967223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775875.223247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775875.479236:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479251:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775875.479256:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479269:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479286:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479294:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775875.479350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664819584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775875.479379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775875.479383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479387:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479393:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775875.479417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664819712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775875.479425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775875.479433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775875.479433:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664819584:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775875.479443:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479445:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479448:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775875.479460:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664819840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775875.479467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775875.479471:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479481:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775875.479490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664819968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775875.479496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479498:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775875.479499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479502:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479505:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479507:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775875.479517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664820096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775875.479521:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775875.479530:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775875.479533:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775875.479534:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479538:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479543:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479547:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775875.479554:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664820224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775875.479561:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775875.479562:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2df80 x1862032664819584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775875.479565:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775875.479569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775875.479577:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17642 00000100:00080000:0.0:1775775875.479582:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775875.479584:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775875.479588:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775875.479593:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664819584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775875.479599:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664820352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775875.479605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479613:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664819584:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775875.479643:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775875.479668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479682:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2ca80 x1862032664819712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664819712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479705:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775875.479714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775875.479731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2f100 x1862032664819840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479747:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664819840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775875.479761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775875.479780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2d180 x1862032664819968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664819968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664819968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775875.479828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775875.479843:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2c380 x1862032664820096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664820096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479863:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775875.479871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664820224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775875.479880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479886:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2f480 x1862032664820224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664820224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479897:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664820224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775875.479906:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664820352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775875.479924:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775875.479931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775875/real 1775775875] req@ffff982974b2fb80 x1862032664820352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775875.479938:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664820352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775875.479943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775875.479945:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775875.479948:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664820352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775875.991175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775876.247292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775877.015195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775877.271312:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775878.039212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775878.295296:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775879.063214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775879.319299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775880.087243:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775880.343221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775880.599182:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599192:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775880.599197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599207:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599221:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599227:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775880.599270:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664820480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775880.599294:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599297:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775880.599299:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599303:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599305:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599309:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775880.599321:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775880.599339:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664820608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775880.599341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664820480:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775880.599360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775880.599365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599371:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599376:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599379:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775880.599384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775880.599395:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664820736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775880.599407:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775880.599410:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599414:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775880.599418:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775880.599424:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775880.599428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775880.599438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664820864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775880.599444:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775880.599449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599452:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599455:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599458:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2e300 x1862032664820480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775775880.599463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775880.599475:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664820992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775880.599482:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599486:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775775880.599486:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664820480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775880.599490:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599497:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599502:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599507:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775880.599508:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775775880.599510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775775880.599515:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775880.599520:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664821120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775880.599523:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664820480:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775880.599535:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775880.599538:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775880.599539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775880.599542:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17648 00000100:00080000:0.0:1775775880.599544:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775880.599547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820608:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775775880.599558:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775880.599570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664821248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775880.599590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2d500 x1862032664820608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599612:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664820608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599622:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599624:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775880.599629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775880.599641:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2e680 x1862032664820736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599655:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664820736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599668:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599672:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775880.599678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775880.599700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599712:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2fb80 x1862032664820864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664820864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775880.599741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775880.599757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2f480 x1862032664820992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664820992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599777:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664820992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775880.599785:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664821120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775880.599794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2c380 x1862032664821120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664821120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599816:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664821120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775880.599835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664821248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775880.599855:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775880.599866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775880/real 1775775880] req@ffff982974b2d180 x1862032664821248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775880.599879:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664821248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775880.599887:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775880.599890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775880.599892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664821248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775881.111196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775881.367285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775882.135181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775882.391202:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775883.159160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775883.415265:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775884.183165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775884.439208:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775885.207150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775885.463253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775885.719231:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719239:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775885.719248:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719259:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719273:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719280:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775885.719323:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664821376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775885.719347:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719349:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775885.719351:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719354:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719357:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719360:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775885.719369:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775885.719380:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664821504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775885.719389:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664821376:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775885.719390:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775885.719400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719404:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719406:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719409:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775885.719412:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775885.719420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664821632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775885.719427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775885.719431:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719446:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775885.719449:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775885.719454:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775885.719460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664821760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775885.719468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775885.719473:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719476:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719480:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719483:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775885.719487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775885.719487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b4e00 x1862032664821376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775885.719499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664821888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775885.719506:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775885.719507:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664821376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775885.719509:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775885.719512:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719520:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719524:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775885.719525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775775885.719529:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775775885.719533:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775775885.719534:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775885.719539:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664821376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775885.719544:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664822016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775885.719550:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775885.719552:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775885.719554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775885.719556:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17653 00000100:00080000:2.0:1775775885.719559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775885.719562:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821504:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775885.719567:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775885.719577:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664822144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775885.719600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b6680 x1862032664821504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664821504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775885.719652:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775885.719668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b4000 x1862032664821632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664821632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719692:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719695:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775885.719699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775885.719710:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b6a00 x1862032664821760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664821760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719731:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775885.719735:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775885.719744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b7100 x1862032664821888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664821888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719762:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719765:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664821888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775885.719769:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775885.719778:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719793:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b7b80 x1862032664822016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719800:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664822016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719805:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719809:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775885.719823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775885.719834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775885.719840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775885/real 1775775885] req@ffff9829764b4e00 x1862032664822144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775885.719847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664822144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775885.719852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775885.719854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775885.719856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775886.231135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775886.487238:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775887.255195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775887.511188:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775888.279205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775888.535233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775889.303232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775889.559245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775890.327128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775890.583235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775890.839147:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775890.839155:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775890.839159:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775890.839173:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839186:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775890.839192:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775890.839232:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664822272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775890.839265:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775890.839270:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775890.839272:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775890.839278:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839283:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775890.839287:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775890.839298:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775890.839314:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664822400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775890.839323:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664822272:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775890.839331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775890.839335:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775890.839337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775890.839342:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775890.839351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775890.839356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775890.839366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664822528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775890.839377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775890.839383:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775890.839383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775890.839386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775890.839391:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839395:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775890.839398:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775890.839402:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775890.839411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664822656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775890.839421:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775890.839426:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9d180 x1862032664822272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775890.839427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775890.839431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775890.839441:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775890.839452:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664822272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775890.839457:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775890.839467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775775890.839469:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839475:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839479:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664822272:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775890.839485:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664822784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775890.839496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775890.839500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775890.839502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775890.839505:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822400:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775890.839510:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:3.0:1775775890.839518:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775890.839523:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775890.839529:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775890.839541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775890.839545:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664822912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775890.839560:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9c000 x1862032664822400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775890.839569:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775890.839573:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664822400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775890.839577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775890.839581:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775890.839590:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775890.839590:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17658 00000100:00080000:0.0:1775775890.839597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775890.839597:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775890.839603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822400:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775890.839606:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775890.839613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822528:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775890.839614:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775890.839634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775890.839639:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664823040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775890.839649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9df80 x1862032664822528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775890.839681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664822528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775890.839692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775890.839706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775890.839724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775890.839732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9c380 x1862032664822656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775890.839740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664822656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775890.839745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775890.839753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775890.839763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775890.839768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9f800 x1862032664822784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775890.839775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664822784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775890.839780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664822784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775890.839797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775890.839807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775890.839814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9c700 x1862032664822912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775890.839821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664822912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775890.839826:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664822912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775890.839834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775890.839843:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775890.839848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775890/real 1775775890] req@ffff982980f9e680 x1862032664823040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775890.839856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664823040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775890.839860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775890.839862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775890.839864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775891.351156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775891.607238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775892.375172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775892.631272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775893.399150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775893.655252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775894.423188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775894.679260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775895.447129:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775895.703264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775895.959119:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959128:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775895.959133:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775895.959143:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959156:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959204:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775895.959243:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664823168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775895.959277:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959279:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775895.959281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775895.959284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959289:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959293:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959303:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775895.959318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664823296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775895.959331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664823168:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775895.959332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775895.959343:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775895.959346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775895.959368:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664823424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775895.959378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775895.959395:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775895.959399:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959403:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959408:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959414:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775895.959426:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664823552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775895.959433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775895.959438:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775895.959442:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823296:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775895.959445:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959451:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959453:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775895.959468:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664823680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775895.959476:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823424:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775895.959477:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775895.959485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775895.959489:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959493:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959496:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959501:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775895.959504:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775895.959513:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664823808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775895.959521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775895.959524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775895.959525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775895.959528:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823680:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775775895.959530:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17663 00000100:00080000:3.0:1775775895.959537:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775895.959540:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775895.959544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775895.959555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664823936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775895.959559:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775895.959586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775896.471183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775896.727278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775897.495133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775897.751247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775898.519173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775898.775277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775899.031481:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775899.031555:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775899.031642:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031700:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031712:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031723:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031732:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031741:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775899.031780:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:0.0:1775775899.032056:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9c380 x1862032664823936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032135:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664823936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032151:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032158:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032164:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775899.032174:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9c700 x1862032664823680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032187:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664823680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032196:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032200:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032204:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775899.032213:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9f800 x1862032664823808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032227:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664823808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032236:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032240:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032243:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664823808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775899.032251:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9fb80 x1862032664823296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032264:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664823296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032272:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032277:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032280:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775899.032288:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9d500 x1862032664823424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032299:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664823424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032307:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032311:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032315:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775899.032322:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9e680 x1862032664823552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032336:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664823552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032345:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032348:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032352:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664823552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775899.032359:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775895/real 1775775899] req@ffff982980f9ed80 x1862032664823168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775899.032374:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664823168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775899.032381:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775899.032386:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775899.032390:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664823168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775899.543203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775899.799251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775900.567216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775900.823295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775901.079239:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079252:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775901.079257:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079269:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079285:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079293:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775901.079345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664824064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775901.079386:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775901.079392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079398:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079402:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079407:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079419:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775901.079435:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664824192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775901.079447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775775901.079456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664824064:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775901.079457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079466:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079474:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079479:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775901.079499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664824320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775901.079508:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079513:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775901.079514:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079519:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079525:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079529:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079536:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775901.079550:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664824448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775901.079560:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775901.079570:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079573:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775901.079578:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079584:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079591:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079596:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079603:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775901.079617:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664824576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775901.079627:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775775901.079629:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2f100 x1862032664824064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775901.079632:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775901.079634:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079645:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079651:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079664:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079682:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775901.079690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664824064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775901.079701:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664824704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775901.079732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.079742:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.079750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664824064:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775901.079764:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775901.079769:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775901.079772:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775901.079778:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17668 00000100:00080000:0.0:1775775901.079784:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775901.079789:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775901.079796:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775901.079805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775901.079815:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664824832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775901.079846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.079867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2ca80 x1862032664824192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.079884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664824192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.079900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.079904:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.079907:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775901.079915:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775901.079941:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.080053:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2df80 x1862032664824320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.080093:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664824320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.080104:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.080107:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.080112:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775901.080122:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775901.080151:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.080164:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2d180 x1862032664824448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.080178:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664824448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.080199:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.080202:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.080206:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775901.080214:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775901.080238:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.080249:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2c380 x1862032664824576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.080260:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664824576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.080268:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.080271:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.080273:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664824576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775901.080280:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664824704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775901.080297:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.080307:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2f480 x1862032664824704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.080325:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664824704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.080333:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.080337:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.080340:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664824704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775901.080365:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664824832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775901.080387:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775901.080398:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775901/real 1775775901] req@ffff982974b2fb80 x1862032664824832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775901.080410:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664824832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775901.080419:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775901.080422:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775901.080425:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664824832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775901.591209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775901.847211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775902.615178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775902.871230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775903.639183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775903.895251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775904.663199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775904.919262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775905.687185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775905.943278:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775906.199212:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199253:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775906.199258:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775906.199266:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199277:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199282:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775906.199315:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664824960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775906.199338:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199341:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775906.199343:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775906.199346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199348:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775906.199357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775906.199366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664825088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775906.199373:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775906.199377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775906.199382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199389:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199394:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664824960:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775906.199397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775906.199414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664825216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775906.199426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775906.199433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775906.199438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199442:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199446:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775906.199453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775906.199460:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775906.199468:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664825344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775906.199480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775906.199486:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775775906.199490:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe3480 x1862032664824960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775906.199495:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199504:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199512:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664824960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775775906.199522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775775906.199524:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199528:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664824960:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775775906.199537:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664825472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775906.199541:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825088:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775906.199557:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775906.199561:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775906.199563:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775906.199568:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00080000:1.0:1775775906.199574:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775775906.199574:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775906.199582:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775906.199589:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775906.199595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe1880 x1862032664825088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775906.199606:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664825600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775906.199614:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664825088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775906.199639:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775906.199644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775906.199645:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775906.199650:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775906.199654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775906.199659:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17673 00000100:00100000:3.0:1775775906.199662:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825088:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775775906.199665:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775906.199672:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775906.199679:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775906.199693:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775775906.199695:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664825728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775906.199717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe2300 x1862032664825216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775906.199754:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664825216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775906.199764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775906.199780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775906.199802:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe2680 x1862032664825344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775906.199825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664825344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775906.199830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775906.199838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775906.199851:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe2a00 x1862032664825472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775906.199864:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664825472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775906.199873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199876:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775906.199881:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664825600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775906.199890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe0380 x1862032664825600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775906.199901:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664825600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775906.199906:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199908:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199910:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664825600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775906.199921:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664825728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775906.199931:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775906.199937:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775906/real 1775775906] req@ffff982981fe0000 x1862032664825728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775906.199943:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664825728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775906.199948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775906.199950:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775906.199952:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664825728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775906.711176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775906.967262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775907.735174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775907.991243:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775908.759194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775909.015234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775909.783197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775910.039269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775910.807161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775911.063247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775911.319240:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319250:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775911.319255:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319265:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319279:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319287:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775911.319334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664825856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775911.319377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775911.319384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319393:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775911.319408:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775911.319424:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664825984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775911.319425:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664825856:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775911.319443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775911.319451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319456:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775911.319471:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775911.319482:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664826112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775911.319492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319496:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775911.319497:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319501:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319505:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319509:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775911.319515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775911.319525:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664826240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775911.319532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775911.319557:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319562:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775911.319565:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319570:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319574:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319579:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775911.319586:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775911.319590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9df80 x1862032664825856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775911.319601:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664826368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775911.319613:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775911.319614:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664825856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775911.319624:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775911.319629:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775911.319637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775911.319638:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319646:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775911.319649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775911.319654:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775911.319657:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664825856:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775775911.319663:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775911.319677:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664826496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775911.319687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825984:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775911.319689:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775911.319696:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775911.319697:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775911.319702:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17678 00000100:00080000:3.0:1775775911.319706:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775911.319711:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775911.319717:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775911.319721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775911.319734:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664826624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775911.319740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9c000 x1862032664825984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.319763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664825984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.319774:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.319779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.319782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664825984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775911.319789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775911.319811:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775911.319823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9d180 x1862032664826112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.319836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664826112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.319845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.319849:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.319852:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775911.319859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775911.319880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775911.319892:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9f480 x1862032664826240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.319904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664826240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.319912:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.319916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.319919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775911.319925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775911.319945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775911.319955:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9ea00 x1862032664826368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.319969:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664826368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.319977:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.319981:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.319984:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775911.320002:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664826496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775911.320023:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775911.320034:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9dc00 x1862032664826496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.320045:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664826496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.320053:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.320056:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.320059:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664826496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775911.320099:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664826624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775911.320123:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775911.320133:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775911/real 1775775911] req@ffff982980f9e300 x1862032664826624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775911.320145:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664826624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775911.320155:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775911.320158:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775911.320162:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664826624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775911.831180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775912.087251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775912.855149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775913.111251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775913.879179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775914.135275:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775914.903207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775915.159265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775915.927201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775916.183291:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775916.439198:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439211:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775916.439216:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439229:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439244:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439251:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775916.439307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664826752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439339:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439342:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775916.439344:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439347:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439351:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439354:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439365:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775916.439378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664826880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439388:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439391:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775916.439393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439397:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439401:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439406:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775916.439421:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664827008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439429:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775916.439431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439433:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439436:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439438:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439442:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775916.439476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664827136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439482:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775916.439486:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439489:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439491:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439493:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439497:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775916.439503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664827264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439510:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439513:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775916.439515:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439517:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439520:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439522:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775916.439529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664827392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775916.439534:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775916.439536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775916.439537:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775916.439541:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17683 00000100:00080000:3.0:1775775916.439543:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775916.439545:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775916.439549:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775916.439558:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664827520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775916.439562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664826752:192.168.201.118@tcp:250:kworker.0 00000800:00000100:0.0:1775775916.439636:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9f100 x1862032664826752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664826752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664826752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775916.439708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775916.439722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9d880 x1862032664826880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664826880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664826880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775916.439758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775916.439769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9ce00 x1862032664827008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439783:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664827008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439788:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775916.439797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775916.439808:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9e300 x1862032664827136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664827136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775916.439834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775916.439844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9dc00 x1862032664827264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439857:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664827264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775916.439870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664827392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775916.439880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439886:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9ea00 x1862032664827392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439893:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664827392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664827392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775916.439926:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664827520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775916.439938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775916.439944:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775916/real 1775775916] req@ffff982980f9f480 x1862032664827520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775916.439951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664827520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775916.439956:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775916.439958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775916.439960:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664827520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775916.951232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775917.207213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775917.975253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775918.231285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775918.999194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775919.255242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775920.023176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775920.279214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775775921.047189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775921.303201:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775921.559205:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559213:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775921.559217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559226:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559238:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775921.559281:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664827648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559312:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775921.559313:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559316:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559318:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559321:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559328:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775921.559336:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664827776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559343:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559346:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775921.559347:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559352:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559373:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559377:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775921.559382:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664827648:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775921.559389:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664827904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559399:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775921.559401:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559403:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559406:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559409:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775921.559421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664828032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559427:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559430:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775921.559432:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559436:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559438:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775921.559445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775921.559450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664828160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775921.559462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559464:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559467:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559470:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775921.559480:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b5c00 x1862032664827648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775921.559481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664828288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775921.559501:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775921.559508:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775921.559510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775921.559512:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17689 00000100:00080000:2.0:1775775921.559515:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775921.559518:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775921.559521:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775921.559526:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664827648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775921.559540:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664828416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775921.559541:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664827648:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775921.559566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775921.559588:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b5f80 x1862032664827776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664827776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559618:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775921.559672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775921.559685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559692:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b4700 x1862032664827904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664827904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559715:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664827904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775921.559720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775921.559731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559737:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b5500 x1862032664828032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664828032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559751:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775921.559756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775921.559766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b4a80 x1862032664828160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664828160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775921.559796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664828288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775921.559805:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b6d80 x1862032664828288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664828288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664828288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775921.559829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664828416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775921.559838:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775921.559843:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775921/real 1775775921] req@ffff9829764b5180 x1862032664828416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775921.559850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664828416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775921.559855:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775921.559857:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775921.559859:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664828416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775922.071213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775922.327289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775923.095186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775923.351261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775924.119133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775924.375256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775925.143182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775925.399248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775926.167203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775926.423253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775926.679223:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679233:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775926.679237:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679248:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679259:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679331:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775926.679366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664828544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775926.679389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775926.679393:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679397:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679400:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679403:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775926.679422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775926.679432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664828672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775926.679440:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664828544:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775926.679443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679449:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775926.679450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679454:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679457:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679462:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775926.679467:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775926.679480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664828800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775926.679490:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775926.679496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679500:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679515:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679520:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775926.679526:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775926.679532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775775926.679541:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664828928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775926.679551:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679555:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775926.679556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679561:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679566:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679570:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775926.679576:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775926.679587:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664829056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775926.679594:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe0000 x1862032664828544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775775926.679597:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775926.679604:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775926.679607:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775926.679615:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679619:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679622:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775926.679626:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775926.679630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664828544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775775926.679634:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664829184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775926.679658:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775926.679659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775775926.679661:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775926.679664:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775926.679668:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775775926.679668:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17694 00000100:00080000:1.0:1775775926.679674:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775926.679678:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775926.679679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664828544:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775775926.679682:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775926.679693:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664829312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775926.679711:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775926.679740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.679754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe0380 x1862032664828672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.679767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664828672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.679775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.679779:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.679783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775926.679789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775926.679810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.679821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe2a00 x1862032664828800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.679833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664828800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.679841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.679844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.679847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775926.679854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775926.679874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.679885:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe2680 x1862032664828928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.679897:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664828928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.679906:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.679909:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.679912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664828928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775926.679918:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775926.679937:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.679948:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe2300 x1862032664829056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.679961:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664829056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.679970:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.679973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.679976:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775926.679982:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664829184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775926.680002:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.680013:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe1880 x1862032664829184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.680026:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664829184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.680035:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.680039:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.680042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664829184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775926.680049:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664829312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775926.680108:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775926.680117:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775926/real 1775775926] req@ffff982981fe3480 x1862032664829312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775926.680125:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664829312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775926.680131:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775926.680133:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775926.680135:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664829312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775927.191200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775927.447235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775928.215180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775928.471290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775929.239208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775929.495256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775930.263165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775930.519244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775931.287221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775931.543305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775931.799226:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799235:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775931.799240:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775931.799250:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799263:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799270:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775931.799311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664829440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799337:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775931.799341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775931.799345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775931.799359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775931.799369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664829568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775931.799381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775931.799383:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664829440:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775931.799386:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799391:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775931.799397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775931.799405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664829696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775931.799412:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775931.799416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775931.799424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775931.799431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664829824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775931.799439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775931.799442:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799450:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775931.799453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775931.799460:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775931.799461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664829952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799475:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775931.799478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775931.799482:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799486:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799490:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775931.799496:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775931.799500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9d180 x1862032664829440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775931.799510:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664830080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775931.799521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775931.799523:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775931.799525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775931.799528:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664829440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775931.799530:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17699 00000100:00080000:3.0:1775775931.799535:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775931.799540:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775931.799542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775931.799544:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775931.799548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664829440:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775931.799561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664830208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775931.799582:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775931.799600:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799609:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9c000 x1862032664829568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799617:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664829568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799626:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775931.799633:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775931.799648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9df80 x1862032664829696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664829696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775931.799673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775931.799685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9ed80 x1862032664829824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664829824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775931.799708:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775931.799717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9e680 x1862032664829952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664829952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664829952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775931.799739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775931.799748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9d500 x1862032664830080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799762:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664830080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775931.799780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775931.799800:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775931.799810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775931/real 1775775931] req@ffff982980f9fb80 x1862032664830208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775931.799819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664830208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775931.799823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775931.799825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775931.799827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775932.311197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775932.567295:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775933.335282:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775933.594490:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775934.359182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775934.615238:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775935.383191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775935.639263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775936.407207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775936.663285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775936.920363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775936.920396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920407:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920456:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920465:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775936.920502:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664830336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775936.920536:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920539:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775936.920540:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920544:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920547:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920550:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775936.920559:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775936.920579:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664830464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775936.920580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664830336:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775936.920595:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920597:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775936.920598:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920602:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920604:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920606:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775936.920609:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775936.920615:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664830592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775936.920622:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920625:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775936.920627:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920629:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920632:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920634:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775936.920637:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775936.920645:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664830720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775936.920651:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920661:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775936.920663:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920665:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920668:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920670:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775936.920674:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775936.920682:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664830848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775936.920689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775936.920695:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920698:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775936.920700:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920703:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00080000:3.0:1775775936.920706:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920709:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775936.920713:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775936.920721:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664830976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775936.920728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9f800 x1862032664830336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775936.920730:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775936.920741:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775936.920745:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775936.920752:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17704 00000100:00100000:0.0:1775775936.920757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664830336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775936.920758:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775936.920768:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775775936.920771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775775936.920775:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775936.920777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.920783:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664830336:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775936.920792:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664831104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775936.920819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775936.920839:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.920856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9c700 x1862032664830464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.920870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664830464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.920877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.920879:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.920881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775936.920885:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775936.920897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.920903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9c380 x1862032664830592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.920911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664830592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.920916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.920919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.920920:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775936.920924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775936.920934:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.920940:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9fb80 x1862032664830720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.920947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664830720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.920956:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.920958:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.920960:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775936.920963:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775936.920972:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.920977:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9d500 x1862032664830848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.920984:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664830848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.920990:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.920991:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.920993:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664830848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775936.920996:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775936.921025:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.921037:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9e680 x1862032664830976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.921049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664830976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.921058:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.921061:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.921113:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664830976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775936.921121:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664831104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775936.921144:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775936.921156:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775936/real 1775775936] req@ffff982980f9ed80 x1862032664831104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775936.921169:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664831104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775936.921178:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775936.921181:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775936.921185:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664831104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775937.431196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775937.687279:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775938.455191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775938.711238:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775939.479191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775939.735245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775940.503184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775940.759250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775941.527159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775941.783263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775942.039206:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039216:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775942.039221:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039232:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039244:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039250:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775942.039288:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664831232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775942.039320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039325:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775942.039327:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039332:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039336:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039341:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775942.039366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664831360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775942.039376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775942.039380:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664831232:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775942.039381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039393:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039399:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775942.039411:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664831488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775942.039418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775942.039422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039428:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039431:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775942.039442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664831616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775942.039451:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775942.039456:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775942.039458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775942.039461:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039466:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039472:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039483:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775942.039495:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b5180 x1862032664831232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775942.039500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664831744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775942.039524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039527:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775942.039528:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775942.039531:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664831232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775942.039534:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17709 00000100:00080000:2.0:1775775942.039541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039554:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775942.039555:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.039563:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775775942.039570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664831872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775942.039571:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664831232:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775942.039608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831360:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775775942.039627:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775942.039631:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775942.039634:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775942.039640:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17709 00000800:00000100:0.0:1775775942.039642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775942.039648:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775942.039655:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775942.039662:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775942.039663:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b6d80 x1862032664831360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775942.039680:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664832000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775942.039683:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664831360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.039712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.039717:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.039722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775942.039729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775942.039752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775942.039763:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b4a80 x1862032664831488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775942.039776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664831488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.039786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.039790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.039794:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775942.039800:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775942.039819:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775942.039830:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b5500 x1862032664831616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775942.039843:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664831616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.039852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.039855:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.039868:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775942.039874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775942.039894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775942.039905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b4700 x1862032664831744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775942.039919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664831744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.039928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.039931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.039934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664831744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775942.039951:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664831872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775942.039970:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775942.039980:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b5f80 x1862032664831872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775942.040013:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664831872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.040023:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.040026:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.040029:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664831872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775942.040036:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775942.040056:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775942.040096:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775942/real 1775775942] req@ffff9829764b5c00 x1862032664832000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775942.040110:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664832000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775942.040119:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775942.040123:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775942.040126:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775942.551176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775942.807297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775943.575138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775943.831250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775944.599137:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775944.855247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775945.623167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775945.879201:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775946.647173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775946.903210:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775775947.159213:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159228:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775775947.159234:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159243:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159254:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159259:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775775947.159293:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664832128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775947.159316:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159319:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775947.159320:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159326:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159328:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159338:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775947.159351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664832256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775947.159363:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664832128:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775775947.159364:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775947.159373:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159378:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159383:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159388:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775947.159409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664832384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775947.159416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775775947.159420:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159423:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159426:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159428:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775947.159439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664832512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775947.159445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775775947.159449:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159451:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159454:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159456:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159459:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775775947.159465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664832640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775947.159473:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775775947.159476:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159479:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775775947.159480:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159483:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159488:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159490:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775947.159496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664832768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775775947.159501:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775775947.159504:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775775947.159505:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775775947.159509:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17714 00000100:00080000:1.0:1775775947.159511:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775775947.159514:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775775947.159518:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775775947.159525:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664832896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775947.159531:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe3b80 x1862032664832128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664832128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159571:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664832128:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775947.159621:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775947.159643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159653:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe1f80 x1862032664832256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664832256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775947.159675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775947.159686:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159692:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe0a80 x1862032664832384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664832384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775947.159738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775947.159760:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe0e00 x1862032664832512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664832512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775947.159799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775947.159814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe1c00 x1862032664832640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664832640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159835:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664832640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775947.159841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775947.159850:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe1500 x1862032664832768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664832768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159868:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159870:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775947.159875:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775947.159886:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775947.159891:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775947/real 1775775947] req@ffff982981fe1180 x1862032664832896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775947.159899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664832896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775947.159904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775947.159905:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775947.159907:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664832896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775947.671159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775947.927267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775948.695282:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775948.951205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775949.719155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775949.975229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775950.743170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775950.999244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775951.767163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775952.023234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775952.279208:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775952.279221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279231:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279244:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279251:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775952.279291:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664833024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775952.279324:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279333:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775952.279335:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279340:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279346:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279351:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775952.279362:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775952.279371:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664833024:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775952.279376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664833152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775952.279385:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279388:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775952.279389:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279395:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279398:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775952.279401:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775952.279409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664833280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775952.279416:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775952.279423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279430:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279434:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775952.279439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775952.279441:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775952.279458:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664833408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775952.279470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775952.279477:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279482:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00100000:3.0:1775775952.279483:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2e680 x1862032664833024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775952.279486:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279492:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775952.279498:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775952.279508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664833536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775952.279519:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279522:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775952.279523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775952.279526:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279530:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775952.279530:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664833024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775952.279537:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775952.279542:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775952.279552:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664833664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775952.279555:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775952.279558:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775952.279561:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775952.279563:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775952.279564:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775952.279568:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17719 00000100:00080000:0.0:1775775952.279572:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775952.279577:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664833024:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775775952.279583:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775952.279591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664833792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775952.279611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775952.279635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2d500 x1862032664833152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664833152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.279659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.279662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.279664:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775952.279668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775952.279680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2e300 x1862032664833280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279714:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664833280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.279729:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.279734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.279738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775952.279745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775952.279766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2fb80 x1862032664833408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664833408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.279801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.279805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.279815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775952.279850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775952.279874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2f480 x1862032664833536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279901:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664833536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.279915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.279918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.279921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664833536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775952.279925:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664833664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775952.279940:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279947:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2c380 x1862032664833664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279955:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664833664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.279960:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.279962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.279964:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664833664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775952.279967:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664833792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775952.279977:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775952.279983:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775952/real 1775775952] req@ffff982974b2d180 x1862032664833792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775952.279994:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664833792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775952.280002:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775952.280005:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775952.280009:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664833792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775952.791189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775953.047245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775953.815161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775954.071213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775954.839199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775955.095202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775955.863203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775956.119236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775956.887203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775957.143261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775957.399183:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399196:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775957.399202:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399215:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399239:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399248:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775957.399305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664833920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775957.399335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775957.399342:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399350:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399356:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775957.399388:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664834048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775957.399400:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664833920:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775957.399404:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399409:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775957.399411:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399416:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399431:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775957.399454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664834176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775957.399463:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399467:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775957.399469:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399478:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399482:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775957.399501:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664834304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775775957.399510:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775775957.399523:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399527:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775957.399529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399536:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399547:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399554:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775957.399555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2df80 x1862032664833920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775957.399574:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664834432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775957.399577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664833920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775957.399635:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775957.399636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775957.399644:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775957.399647:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775957.399650:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775957.399655:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00100000:3.0:1775775957.399661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664833920:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775957.399662:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399672:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399677:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775957.399690:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664834560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775957.399699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775957.399705:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775957.399709:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775957.399712:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775957.399719:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17724 00000100:00080000:0.0:1775775957.399726:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775957.399745:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775957.399748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775957.399760:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775957.399777:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664834688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775957.399780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2ca80 x1862032664834048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.399829:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664834048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.399839:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.399843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.399850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775957.399858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775957.399882:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775957.399894:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2f100 x1862032664834176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.399906:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664834176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.399914:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.399918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.399921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775957.399927:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775957.399950:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775957.399960:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2d180 x1862032664834304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.399969:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664834304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.399975:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.399998:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.400002:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775957.400009:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775957.400034:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775957.400047:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2c380 x1862032664834432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.400061:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664834432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.400098:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.400103:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.400107:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775957.400131:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664834560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775957.400156:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775957.400169:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2f480 x1862032664834560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.400182:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664834560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.400191:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.400195:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.400198:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664834560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775957.400205:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664834688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775957.400229:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775957.400241:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775957/real 1775775957] req@ffff982974b2fb80 x1862032664834688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775957.400255:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664834688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775957.400265:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775957.400269:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775957.400272:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664834688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775957.911187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775958.167357:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775958.935201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775959.191222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775959.959181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775960.215244:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775960.983206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775961.239271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775962.007179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775962.263240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775962.519136:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519143:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775962.519147:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519156:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519173:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519178:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775962.519214:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664834816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519239:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519242:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775962.519243:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519248:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519251:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519253:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519261:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775962.519269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664834944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519275:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519277:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775962.519278:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519281:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519282:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519284:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519287:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775962.519293:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664835072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519299:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519301:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775962.519302:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519305:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519307:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519310:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519313:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775962.519317:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664834816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775962.519323:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664835200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775962.519335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519338:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519339:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519342:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775962.519351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664835328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519357:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775962.519360:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519362:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519365:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519367:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775962.519375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664835456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775962.519380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775962.519382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775962.519383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775962.519386:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17730 00000100:00080000:0.0:1775775962.519389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775962.519392:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775962.519396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775962.519406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664835584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775962.519425:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775962.519449:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775962.519472:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775962.519493:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775962.519511:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664835456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775962.519525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664835584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775963.031167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775963.287227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775964.055195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775964.311252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775965.079167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775965.335254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775775965.591372:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775775965.591415:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775775965.591434:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591479:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591489:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591497:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591507:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591516:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775775965.591529:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775775965.591543:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2fb80 x1862032664835200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591580:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664835200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591631:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775965.591645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2f480 x1862032664835328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664835328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775965.591680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2c380 x1862032664835456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664835456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664835456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775965.591710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2d180 x1862032664835584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664835584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664835584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775965.591740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2e300 x1862032664834816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664834816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591757:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664834816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775965.591769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2d500 x1862032664834944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664834944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664834944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775965.591799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775962/real 1775775965] req@ffff982974b2e680 x1862032664835072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775965.591811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664835072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775965.591819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775965.591822:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775965.591825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775966.103239:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775966.359280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775967.127169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775967.383243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775775967.639182:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639192:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775775967.639197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639219:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639226:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775967.639257:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664835712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775967.639281:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639285:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775967.639287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639291:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639295:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639299:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775967.639322:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664835840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775967.639330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664835712:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775967.639334:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775967.639339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639346:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775967.639364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664835968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775967.639372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775775967.639377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639380:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639383:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639386:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775775967.639392:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775967.639401:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664836096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775967.639409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775775967.639415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775775967.639422:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2f100 x1862032664835712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775967.639425:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775967.639438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664836224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775967.639440:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664835712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775967.639449:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775967.639451:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775967.639452:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775775967.639454:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775967.639455:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775775967.639458:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00100000:3.0:1775775967.639459:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664835712:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775775967.639461:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639464:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639468:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775967.639477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664836352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775967.639482:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835840:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775775967.639487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775775967.639490:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775967.639492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775967.639495:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17735 00000100:00080000:0.0:1775775967.639499:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775967.639503:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775775967.639504:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775775967.639508:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775967.639516:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2ca80 x1862032664835840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775967.639517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664836480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775775967.639537:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664835840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639545:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639585:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775967.639593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775967.639625:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775967.639638:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2df80 x1862032664835968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775967.639650:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664835968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639664:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664835968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775967.639670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775967.639692:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775967.639704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2e680 x1862032664836096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775967.639715:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664836096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775967.639740:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775775967.639761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775967.639773:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2d500 x1862032664836224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775967.639785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664836224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639798:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775775967.639816:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664836352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775967.639837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775967.639848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2e300 x1862032664836352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775967.639859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664836352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639870:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639874:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664836352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775967.639880:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664836480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775775967.639900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775967.639909:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775967/real 1775775967] req@ffff982974b2d180 x1862032664836480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775775967.639920:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664836480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775967.639926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775967.639930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775775967.639932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664836480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775968.151130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775968.407236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775969.175142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775969.431273:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775970.199222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775970.455275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775971.223148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775971.479283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775972.247193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775972.503281:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775972.759222:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775972.759233:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775972.759237:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759247:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:3.0:1775775972.759260:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759267:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775972.759307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664836608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775972.759345:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775972.759348:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775972.759349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759353:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:3.0:1775775972.759356:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759359:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759367:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775972.759377:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664836736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775972.759385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775972.759389:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664836608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775972.759389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775972.759392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759399:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:3.0:1775775972.759402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759404:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775972.759416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664836864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775972.759422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775972.759425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775972.759426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:3.0:1775775972.759431:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759437:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775972.759444:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664836992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775972.759450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775972.759455:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775972.759456:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775972.759458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759476:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:3.0:1775775972.759492:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759495:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759499:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775972.759508:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664837120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775972.759516:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775972.759518:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775972.759520:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759524:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00100000:0.0:1775775972.759525:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9df80 x1862032664836608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775972.759527:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775972.759533:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775972.759549:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664837248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775972.759557:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664836608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775972.759574:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775972.759578:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775775972.759580:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775972.759580:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775972.759588:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17740 00000100:00080000:0.0:1775775972.759589:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775972.759592:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775972.759595:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664836608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775972.759599:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775972.759604:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775972.759611:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664837376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775972.759623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775972.759645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9c000 x1862032664836736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664836736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759673:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759680:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775972.759686:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775972.759706:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9d180 x1862032664836864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664836864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775972.759749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775972.759782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9f480 x1862032664836992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664836992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664836992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775972.759826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775972.759842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9ea00 x1862032664837120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759856:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664837120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759868:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759870:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775972.759873:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664837248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775972.759883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9dc00 x1862032664837248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664837248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759900:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664837248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775972.759907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664837376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775972.759916:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775972.759922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775972/real 1775775972] req@ffff982980f9e300 x1862032664837376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775972.759928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664837376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775775997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775972.759933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775972.759935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775972.759937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664837376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775973.271144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775973.527218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775974.295189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775974.551261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775975.319211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775975.575230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775976.343154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775775976.599255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775977.367134:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775977.623231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775977.879214:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879227:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775977.879232:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879245:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879267:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879275:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775977.879318:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664837504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775977.879341:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775977.879346:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879349:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879351:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879353:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775977.879384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664837632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775977.879397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775977.879402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775977.879404:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664837504:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775977.879406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879415:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879419:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775977.879428:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664837760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775977.879433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775977.879437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879439:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879441:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879443:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879446:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775977.879456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664837888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775977.879461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775977.879466:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879469:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879480:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775977.879486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664838016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775977.879491:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775977.879496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775977.879500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879502:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879506:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879508:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879510:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775977.879516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664838144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775977.879521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775977.879523:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775977.879524:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775977.879528:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17745 00000100:00080000:2.0:1775775977.879530:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775977.879533:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775977.879535:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775977.879542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664838272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775977.879551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b4e00 x1862032664837504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879575:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664837504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879587:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879592:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879596:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664837504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775977.879625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775977.879648:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879661:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b7b80 x1862032664837632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879674:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664837632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775977.879699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775977.879719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b7100 x1862032664837760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664837760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879759:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879765:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775977.879771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775977.879791:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b6a00 x1862032664837888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664837888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664837888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775977.879832:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838016:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775977.879858:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879869:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b4000 x1862032664838016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664838016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879890:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879893:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879896:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775977.879902:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664838144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775977.879921:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b6680 x1862032664838144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879942:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664838144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.879950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.879953:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.879956:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664838144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775977.879961:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664838272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775977.879979:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775977.879988:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775977/real 1775775977] req@ffff9829764b5c00 x1862032664838272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775977.879999:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664838272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775977.880008:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775977.880012:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775977.880015:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664838272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775978.391194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775978.647296:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775979.415171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775979.671337:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775980.439171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775980.695320:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775981.463232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775981.719302:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775982.487180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775982.743263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775982.999236:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775982.999250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999261:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999274:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999281:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775982.999338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664838400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775982.999366:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999369:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775982.999370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999373:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999376:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999379:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999390:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775982.999404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664838528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775982.999415:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999419:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775982.999421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999425:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999429:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999434:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775982.999454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664838656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775982.999457:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664838400:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775982.999466:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999472:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775982.999474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999478:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999480:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775982.999495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664838784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775982.999502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999506:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775982.999507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999515:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775982.999527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664838912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775982.999532:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999535:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775982.999537:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999541:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999543:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999545:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775982.999556:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775775982.999559:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664839040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775982.999577:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775982.999580:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775982.999582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775982.999585:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17750 00000100:00080000:2.0:1775775982.999589:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775982.999592:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775982.999596:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775982.999606:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b5f80 x1862032664838400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775982.999606:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664839168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775982.999695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664838400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775982.999704:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775982.999708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775982.999711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664838400:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775775982.999718:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775982.999739:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775982.999749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b4700 x1862032664838528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775982.999757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664838528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775982.999767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775982.999770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775982.999772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775982.999795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775982.999816:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775982.999828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b5500 x1862032664838656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775982.999841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664838656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775982.999851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775982.999855:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775982.999858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775982.999865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775982.999898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775982.999911:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b4a80 x1862032664838784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775982.999925:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664838784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775982.999931:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775982.999933:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775982.999936:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775982.999940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775982.999957:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775982.999964:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b6d80 x1862032664838912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775982.999971:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664838912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775982.999977:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775982.999979:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775982.999981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664838912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775982.999985:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775982.999995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775983.000000:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775982] req@ffff9829764b5180 x1862032664839040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775983.000008:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664839040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775983.000016:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775983.000019:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775983.000022:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775983.000028:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775983.000063:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775983.000112:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775982/real 1775775983] req@ffff9829764b5c00 x1862032664839168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775983.000120:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664839168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775983.000126:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775983.000128:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775983.000130:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775983.511214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775983.767290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775984.535153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775984.791271:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775985.559228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775985.815263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775986.583190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775986.839318:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775987.607218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775987.863308:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775988.119443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775988.119463:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119476:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:3.0:1775775988.119490:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775988.119499:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775988.119548:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664839296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775988.119582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119587:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775988.119589:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119594:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:3.0:1775775988.119598:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775988.119602:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775988.119614:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775988.119629:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664839424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775988.119640:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119644:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775988.119646:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119652:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:3.0:1775775988.119656:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775988.119662:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775988.119669:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775988.119670:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664839296:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775988.119694:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664839552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775988.119717:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119721:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775988.119723:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119728:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:3.0:1775775988.119733:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775988.119739:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775988.119745:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775775988.119750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775988.119781:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664839680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775988.119792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9ce00 x1862032664839296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.119814:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119820:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775775988.119824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664839296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.119826:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119842:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:0.0:1775775988.119843:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.119852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775988.119852:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775988.119858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664839296:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775988.119864:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775988.119870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839424:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775988.119872:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775988.119893:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664839808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775988.119894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775988.119905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9d880 x1862032664839424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.119910:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.119925:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775988.119927:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775988.119932:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00100000:0.0:1775775988.119937:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664839424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.119939:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775988.119952:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.119955:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775988.119958:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839424:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775988.119960:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775988.119970:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775988.119984:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775988.119990:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664839936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775988.120004:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775988.120006:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775988.120011:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775775988.120014:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9f100 x1862032664839552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.120016:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775988.120032:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664839552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775988.120041:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17755 00000100:00080000:0.0:1775775988.120043:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.120053:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775988.120053:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775988.120062:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839552:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775775988.120110:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775988.120117:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775988.120135:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664840064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775988.120147:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775988.120171:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775988.120181:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9e300 x1862032664839680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775988.120190:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664839680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775988.120220:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.120224:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775988.120228:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775988.120235:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839808:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775988.120276:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775988.120291:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9dc00 x1862032664839808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775988.120303:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664839808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775988.120312:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.120315:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775988.120318:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664839808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775988.120335:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775988.120356:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775988.120368:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9ea00 x1862032664839936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775988.120379:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664839936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775988.120387:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.120391:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775988.120395:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664839936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775988.120400:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775988.120421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775988.120432:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775988/real 1775775988] req@ffff982980f9f480 x1862032664840064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775988.120443:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664840064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775988.120451:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775988.120455:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775988.120458:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775988.631233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775988.887257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775989.655203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775989.911279:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775990.679144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775990.935251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775991.703188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775991.959292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775992.727202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775992.983243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775775993.239210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775775993.239226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775993.239237:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239251:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775993.239259:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775775993.239299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664840192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775993.239332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775993.239338:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775993.239343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775993.239352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775993.239363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775993.239375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664840192:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775775993.239382:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664840320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775993.239395:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775993.239401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775993.239406:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775993.239415:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775993.239421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775775993.239432:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775775993.239437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664840448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775993.239450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775775993.239456:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775993.239462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239468:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775993.239474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9d180 x1862032664840192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775993.239474:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775775993.239484:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775993.239501:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664840576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775993.239504:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664840192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239517:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239526:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239533:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664840192:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775775993.239534:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239543:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775775993.239548:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775775993.239554:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239559:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775993.239564:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239568:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840320:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775993.239570:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775775993.239586:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664840704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775993.239595:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239607:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9c000 x1862032664840320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775993.239614:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775993.239619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664840320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775775993.239621:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775775993.239626:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775775993.239633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775775993.239636:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239641:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775775993.239649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775775993.239650:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840320:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775775993.239659:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775993.239671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775993.239677:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664840832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775775993.239690:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775775993.239695:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775775993.239697:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775775993.239702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775775993.239706:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17760 00000100:00080000:3.0:1775775993.239712:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775775993.239717:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9df80 x1862032664840448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775775993.239724:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775775993.239739:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664840960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775993.239744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664840448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775993.239779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775993.239798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9ed80 x1862032664840576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775993.239814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664840576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239822:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239824:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775993.239839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775993.239857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9e680 x1862032664840704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775993.239881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664840704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239889:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239892:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239894:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664840704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775993.239899:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775993.239913:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239920:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9d500 x1862032664840832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775993.239928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664840832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775993.239940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775993.239949:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775993.239955:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775993/real 1775775993] req@ffff982980f9fb80 x1862032664840960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775993.239961:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664840960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775993.239967:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775993.239969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775993.239970:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664840960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775993.751201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775994.007299:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775994.775193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775995.031205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775995.799178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775996.055252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775996.823168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775997.079254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775997.847166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775775998.103233:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775775998.359183:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775998.359193:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775775998.359198:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359208:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359222:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775998.359229:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775775998.359271:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664841088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775998.359302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775998.359306:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775998.359308:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359313:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359318:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775998.359323:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775998.359335:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775775998.359345:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664841088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775775998.359353:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664841216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775998.359365:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775998.359369:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775998.359371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359376:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359380:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775998.359385:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775998.359390:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775775998.359404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664841344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775998.359406:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775998.359418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775998.359422:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775775998.359425:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359431:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359436:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775998.359440:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b6680 x1862032664841088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775998.359444:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775998.359454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775775998.359462:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664841088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775775998.359470:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664841472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775775998.359484:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359492:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775775998.359492:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775775998.359498:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664841088:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775775998.359499:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775775998.359503:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359509:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359514:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775998.359518:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359522:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841216:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775775998.359526:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775775998.359541:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664841600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775775998.359554:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775998.359555:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775775998.359561:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775775998.359563:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775775998.359568:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00100000:0.0:1775775998.359570:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b4000 x1862032664841216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775998.359573:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775775998.359581:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359585:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664841216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775775998.359586:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775998.359596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359599:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775998.359608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775998.359612:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664841728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775775998.359624:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775775998.359625:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775775998.359631:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775775998.359633:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775775998.359636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b6a00 x1862032664841344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775998.359641:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17765 00000100:00080000:2.0:1775775998.359649:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775775998.359655:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664841344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775775998.359656:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775775998.359665:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775775998.359666:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359672:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359675:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775775998.359681:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664841856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775775998.359683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775998.359704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b7100 x1862032664841472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775998.359721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664841472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775998.359726:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359728:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359730:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775998.359744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775775998.359763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b7b80 x1862032664841600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775998.359787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664841600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775998.359796:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359799:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664841600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775775998.359809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664841728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775998.359825:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359833:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b4e00 x1862032664841728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775998.359840:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664841728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775998.359846:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359848:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664841728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775775998.359853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664841856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775775998.359863:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775775998.359868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775775998/real 1775775998] req@ffff9829764b6680 x1862032664841856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775775998.359876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664841856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775775998.359880:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775775998.359882:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775775998.359884:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664841856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775775998.871210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775775999.127249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775775999.895176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776000.151218:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776000.919168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776001.175234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776001.943182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776002.199245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776002.967179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776003.223261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776003.479176:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479185:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776003.479188:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479197:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479207:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479212:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776003.479247:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664841984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776003.479269:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479272:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776003.479273:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479276:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479278:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479281:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776003.479302:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664842112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776003.479303:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664841984:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776003.479317:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479319:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776003.479320:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479323:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479326:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479328:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479331:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776003.479339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664842240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776003.479345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479348:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776003.479349:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479351:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479354:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479356:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479360:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776003.479367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664842368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776003.479373:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776003.479379:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776003.479384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479388:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479394:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479396:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479400:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776003.479409:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b5c00 x1862032664841984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776003.479409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664842496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776003.479436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776003.479439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776003.479440:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479443:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479445:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479447:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776003.479455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664842624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776003.479472:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664841984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479480:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479489:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776003.479491:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776003.479494:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664841984:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776003.479496:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776003.479500:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776003.479505:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17770 00000100:00080000:2.0:1775776003.479508:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776003.479511:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776003.479514:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776003.479521:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775776003.479527:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664842752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776003.479550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b5180 x1862032664842112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664842112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479598:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776003.479612:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776003.479635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b6d80 x1862032664842240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664842240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776003.479675:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776003.479689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b4a80 x1862032664842368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479703:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664842368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776003.479715:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842496:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776003.479726:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b5500 x1862032664842496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479739:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664842496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479743:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479747:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664842496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776003.479751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664842624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776003.479761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b4700 x1862032664842624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479774:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664842624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664842624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776003.479809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664842752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776003.479830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776003.479842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776003/real 1775776003] req@ffff9829764b5f80 x1862032664842752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776003.479854:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664842752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776003.479861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776003.479865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776003.479869:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664842752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776003.991199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776004.247229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776005.015217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776005.271229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776006.039143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776006.295268:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776007.063201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776007.319262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776008.087157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776008.343259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776008.599232:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776008.599259:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776008.599268:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776008.599283:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599300:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599309:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776008.599365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664842880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776008.599405:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776008.599410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776008.599412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776008.599417:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599422:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599427:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776008.599455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664843008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776008.599467:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776008.599472:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664842880:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776008.599472:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776008.599475:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776008.599486:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599490:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599496:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599502:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776008.599516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664843136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776008.599527:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776008.599532:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776008.599534:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776008.599538:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599542:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599546:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599553:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775776008.599567:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776008.599568:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664843264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776008.599581:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776008.599585:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776008.599588:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776008.599593:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599598:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599602:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599608:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776008.599619:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b5f80 x1862032664842880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776008.599623:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664843392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776008.599656:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776008.599661:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776008.599663:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776008.599667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664842880/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776008.599669:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599672:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599677:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599683:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776008.599695:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664843520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776008.599697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776008.599708:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776008.599709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776008.599715:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776008.599717:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776008.599719:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664842880:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776008.599723:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17776 00000100:00080000:2.0:1775776008.599728:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776008.599733:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776008.599739:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776008.599750:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664843648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776008.599765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776008.599789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.599800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b4700 x1862032664843008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.599817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664843008/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.599823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.599826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.599829:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776008.599835:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776008.599872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.599884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b5500 x1862032664843136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.599907:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664843136/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.599914:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.599916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.599919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776008.599926:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776008.599958:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.599970:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b4a80 x1862032664843264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.599983:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664843264/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.599989:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.599992:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.600001:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776008.600006:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843392:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776008.600023:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.600033:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b6d80 x1862032664843392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.600053:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664843392/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.600117:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.600122:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.600126:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776008.600133:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664843520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776008.600156:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.600165:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b5180 x1862032664843520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.600177:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664843520/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.600186:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.600190:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.600193:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664843520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776008.600200:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664843648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776008.600222:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776008.600233:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776008/real 1775776008] req@ffff9829764b5c00 x1862032664843648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776008.600241:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664843648/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776008.600247:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776008.600249:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776008.600252:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664843648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776009.111201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776009.367284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776010.135172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776010.391157:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776011.159177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776011.415184:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776012.183176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776012.439174:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776013.207205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776013.463264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776013.719229:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776013.719250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719262:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719277:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719285:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776013.719334:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664843776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776013.719370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776013.719375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719380:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719383:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719387:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776013.719407:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664843904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776013.719413:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719416:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776013.719417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719420:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719422:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719424:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719429:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776013.719430:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664843776:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776013.719440:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664844032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776013.719450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776013.719455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719472:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776013.719486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664844160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776013.719495:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719499:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776013.719501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719505:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719509:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719515:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775776013.719516:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719521:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776013.719534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664844288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776013.719544:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719547:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776013.719549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776013.719553:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00080000:3.0:1775776013.719557:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776013.719562:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9c380 x1862032664843776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776013.719563:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719571:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776013.719582:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664843776/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776013.719586:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664844416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776013.719611:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776013.719611:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776013.719618:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776013.719620:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776013.719621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776013.719627:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17781 00000100:00100000:0.0:1775776013.719630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664843776:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776013.719631:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776013.719637:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776013.719642:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776013.719645:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776013.719654:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664844544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776013.719687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.719700:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9c700 x1862032664843904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.719724:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664843904/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.719733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.719736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.719739:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664843904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776013.719778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776013.719798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.719807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9f800 x1862032664844032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.719819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664844032/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.719827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.719830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.719833:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776013.719847:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844160:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776013.719867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.719877:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9fb80 x1862032664844160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.719887:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664844160/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.719896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.719899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.719902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776013.719908:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776013.719928:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.719940:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9d500 x1862032664844288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.719956:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664844288/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.719964:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.719967:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.719970:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776013.719975:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664844416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776013.719995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.720006:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9e680 x1862032664844416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.720018:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664844416/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.720025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.720028:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.720031:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664844416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776013.720036:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664844544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776013.720056:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776013.720108:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776013/real 1775776013] req@ffff982980f9ed80 x1862032664844544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776013.720121:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664844544/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776013.720129:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776013.720132:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776013.720135:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664844544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776014.231202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776014.487193:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776015.255182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776015.511312:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776016.279144:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776016.535248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776017.303141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776017.559278:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776018.327157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776018.583282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776018.839192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776018.839201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776018.839206:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839216:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839229:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839235:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776018.839291:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664844672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776018.839321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776018.839324:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776018.839326:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839337:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839348:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776018.839371:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664844800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776018.839375:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664844672:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776018.839397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776018.839400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776018.839401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839405:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839415:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776018.839444:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664844928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776018.839455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775776018.839458:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776018.839463:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776018.839466:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839470:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839475:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839479:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776018.839497:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664845056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776018.839504:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776018.839510:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b6680 x1862032664844672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776018.839515:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776018.839517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839528:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839535:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839540:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839543:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776018.839549:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664844672/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776018.839557:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664845184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776018.839560:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776018.839578:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776018.839582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776018.839583:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776018.839585:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776018.839591:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00100000:0.0:1775776018.839595:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664844672:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776018.839595:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839607:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839611:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776018.839620:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664845312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776018.839627:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776018.839629:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776018.839630:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776018.839635:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844800:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776018.839638:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17786 00000100:00080000:2.0:1775776018.839641:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776018.839647:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776018.839650:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775776018.839688:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776018.839694:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664845440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776018.839706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b4e00 x1862032664844800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.839723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664844800/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.839734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.839738:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.839742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776018.839748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776018.839766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776018.839774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b7b80 x1862032664844928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.839784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664844928/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.839792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.839795:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.839797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664844928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776018.839803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776018.839824:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776018.839834:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b7100 x1862032664845056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.839846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664845056/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.839853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.839856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.839860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776018.839866:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776018.839886:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776018.839897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b6a00 x1862032664845184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.839908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664845184/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.839916:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.839919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.839922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776018.839928:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664845312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776018.839946:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776018.839956:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b4000 x1862032664845312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.839966:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664845312/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.839974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.839977:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.839980:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664845312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776018.840001:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664845440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776018.840019:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776018.840028:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776018/real 1775776018] req@ffff9829764b6680 x1862032664845440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776018.840039:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664845440/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776018.840047:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776018.840050:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776018.840053:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664845440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776019.351207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776019.607261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776020.375177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776020.631260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776021.399156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776021.655213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776022.423178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776022.679290:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776023.447161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776023.703240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776023.959158:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959169:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776023.959173:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959183:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959204:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959211:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776023.959245:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664845568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959264:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959267:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776023.959268:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959272:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959274:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959276:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959283:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776023.959305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664845696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776023.959320:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664845568:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776023.959321:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776023.959324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959331:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959334:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959337:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776023.959350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664845824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959356:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959358:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776023.959359:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959361:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959364:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959365:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959368:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776023.959374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664845952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776023.959382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959386:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17791 00000800:00000100:0.0:1775776023.959390:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776023.959390:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959395:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776023.959403:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664846080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776023.959412:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959415:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959418:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776023.959420:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776023.959426:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9df80 x1862032664845568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776023.959431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664846208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776023.959452:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776023.959454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776023.959455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776023.959458:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17791 00000100:00080000:3.0:1775776023.959460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776023.959463:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664845568/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776023.959465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776023.959469:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776023.959477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664846336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776023.959480:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959487:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959492:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664845568:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776023.959517:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776023.959541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959553:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9c000 x1862032664845696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959566:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664845696/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959585:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959589:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959592:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776023.959599:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776023.959620:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959631:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9d180 x1862032664845824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959643:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664845824/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959653:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959657:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776023.959662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776023.959682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959693:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9f480 x1862032664845952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664845952/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959712:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959719:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664845952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776023.959725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776023.959741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9ea00 x1862032664846080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664846080/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776023.959775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664846208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776023.959794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9dc00 x1862032664846208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664846208/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959821:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959827:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664846208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776023.959833:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664846336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776023.959849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776023.959857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776023/real 1775776023] req@ffff982980f9e300 x1862032664846336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776023.959867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664846336/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776023.959874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776023.959875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776023.959877:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664846336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776024.471168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776024.727235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776025.495161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776025.751200:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776026.519194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776026.775273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776027.543149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776027.799225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776028.567165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776028.823229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776029.079152:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079165:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776029.079170:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079183:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079203:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079214:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776029.079257:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664846464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776029.079279:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079282:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776029.079287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079293:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079295:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079299:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776029.079309:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776029.079318:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664846592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776029.079326:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664846464:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776029.079328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079333:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776029.079334:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079338:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079340:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079342:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776029.079344:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776029.079351:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664846720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776029.079370:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079374:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776029.079376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079384:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079388:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776029.079394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776029.079408:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664846848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776029.079417:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776029.079423:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079428:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079432:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079438:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776029.079440:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846592:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775776029.079446:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776029.079462:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664846976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776029.079470:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846720:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776029.079475:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079478:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776029.079495:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776029.079502:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846848:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776029.079502:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079509:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079514:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776029.079519:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776029.079535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776029.079535:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664847104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776029.079550:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776029.079554:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775776029.079557:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664847104:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776029.079557:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776029.079565:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17796 00000100:00080000:3.0:1775776029.079569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776029.079572:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776029.079577:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776029.079588:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664847232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776029.079614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664847232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776029.591163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776029.847252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776030.615191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776030.871247:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776031.639188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776031.895197:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775776032.151348:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:0.0:1775776032.151408:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775776032.151707:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151776:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151787:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151796:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151805:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151815:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776032.151823:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776032.151945:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9f480 x1862032664847232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.151978:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664847232/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.151992:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.151999:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152003:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664847232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776032.152013:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9ea00 x1862032664847104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152451:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664847104/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152462:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152467:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152470:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664847104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776032.152478:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9dc00 x1862032664846976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152491:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664846976/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152498:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152502:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152505:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776032.152512:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9d880 x1862032664846592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152523:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664846592/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152531:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152538:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776032.152545:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9ce00 x1862032664846720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152555:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664846720/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152562:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152565:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152568:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776032.152574:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9e300 x1862032664846848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664846848/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664846848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776032.152603:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776029/real 1775776032] req@ffff982980f9f100 x1862032664846464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776032.152615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664846464/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776032.152622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776032.152625:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776032.152627:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664846464:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776032.663231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776032.919280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776033.687162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776033.943167:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776034.199168:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776034.199177:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776034.199180:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199190:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199200:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776034.199206:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776034.199237:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664847360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776034.199262:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776034.199266:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776034.199267:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199271:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199276:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776034.199281:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776034.199291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776034.199302:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664847360:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776034.199305:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664847488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776034.199319:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776034.199322:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776034.199324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199329:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199338:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776034.199342:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776034.199347:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776034.199358:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664847616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776034.199368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776034.199369:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776034.199374:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776034.199377:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199382:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199404:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776034.199414:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199417:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b5c00 x1862032664847360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776034.199421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776034.199439:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664847744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776034.199444:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664847360/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776034.199464:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776034.199466:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776034.199472:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776034.199475:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776034.199476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776034.199482:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664847360:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776034.199485:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199492:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776034.199496:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776034.199502:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776034.199513:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664847872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776034.199515:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847488:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776034.199532:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776034.199536:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776034.199538:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199543:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199546:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776034.199549:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199562:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b5180 x1862032664847488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776034.199565:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776034.199574:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776034.199579:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664847488/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776034.199588:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776034.199589:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664848000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776034.199594:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.199602:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847488:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776034.199607:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776034.199611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847616:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776034.199613:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776034.199617:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776034.199623:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17801 00000100:00080000:2.0:1775776034.199633:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776034.199633:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776034.199642:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b6d80 x1862032664847616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776034.199654:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776034.199659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664847616/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776034.199674:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664848128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776034.199674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776034.199682:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.199689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776034.199698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776034.199716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199725:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b4a80 x1862032664847744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776034.199735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664847744/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776034.199741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776034.199743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.199745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776034.199758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776034.199775:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199784:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b5500 x1862032664847872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776034.199794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664847872/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776034.199801:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776034.199804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.199807:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664847872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776034.199813:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776034.199831:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199839:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b4700 x1862032664848000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776034.199849:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664848000/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776034.199856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776034.199859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.199862:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776034.199868:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776034.199885:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776034.199993:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776034/real 1775776034] req@ffff9829764b5f80 x1862032664848128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776034.200005:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664848128/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776034.200013:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776034.200016:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776034.200019:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776034.711159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776034.967241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776035.735206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776035.991266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776036.759196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776037.015272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776037.783202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776038.039363:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776038.807228:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776039.064370:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776039.319230:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319239:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776039.319248:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319260:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319276:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319291:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776039.319362:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664848256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776039.319394:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319398:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776039.319400:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319404:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319413:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319417:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319427:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776039.319438:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664848384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776039.319449:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319453:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775776039.319454:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664848256:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776039.319454:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319459:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319465:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319469:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319474:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776039.319485:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664848512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776039.319494:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776039.319499:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319503:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319507:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319511:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776039.319528:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664848640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776039.319535:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775776039.319539:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776039.319539:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776039.319541:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319546:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319548:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319550:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319554:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776039.319561:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664848768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776039.319568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776039.319572:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319576:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319579:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319582:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776039.319588:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664848896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776039.319594:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9f100 x1862032664848256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776039.319594:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776039.319598:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776039.319601:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776039.319608:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17806 00000100:00080000:3.0:1775776039.319612:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776039.319615:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776039.319617:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776039.319624:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664849024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776039.319627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664848256/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.320755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.320760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.320763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664848256:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776039.320789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776039.320816:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.320828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9e300 x1862032664848384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.320836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664848384/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.320841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.320843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.320845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776039.320848:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776039.320859:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.320864:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9ce00 x1862032664848512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.320871:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664848512/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.320875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.320877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.321840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776039.321848:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776039.321875:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.321884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9d880 x1862032664848640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.321892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664848640/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.321898:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.321901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.321904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776039.321910:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776039.321929:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.321939:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9dc00 x1862032664848768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.321951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664848768/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.321959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.321962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.321963:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664848768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776039.321968:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776039.321984:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.321992:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9ea00 x1862032664848896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.322000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664848896/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.322008:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.322011:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.322014:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664848896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776039.322019:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776039.322039:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776039.322049:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776039/real 1775776039] req@ffff982980f9f480 x1862032664849024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776039.322058:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664849024/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776039.322063:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776039.322083:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776039.322086:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776039.831198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776040.087304:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776040.855188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776041.111269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776041.879235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776042.135293:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776042.903146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776043.159263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776043.927168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776044.183288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776044.439172:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439179:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776044.439183:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439191:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439202:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439207:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776044.439259:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664849152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439285:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439288:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776044.439290:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439295:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439302:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439309:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664849152:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775776044.439310:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776044.439324:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664849280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439331:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776044.439334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439337:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439340:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439342:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776044.439344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776044.439351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664849408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776044.439364:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439366:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439368:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439370:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776044.439373:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775776044.439376:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776044.439384:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664849536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439392:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776044.439394:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439396:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439398:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439400:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776044.439403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776044.439408:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9d180 x1862032664849152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776044.439410:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664849664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776044.439435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776044.439438:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664849152/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776044.439441:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439446:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439451:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:0.0:1775776044.439453:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775776044.439454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775776044.439458:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439463:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664849152:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776044.439463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664849792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776044.439472:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776044.439474:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776044.439475:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776044.439477:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17811 00000100:00080000:3.0:1775776044.439479:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776044.439481:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776044.439484:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776044.439485:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776044.439491:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664849920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776044.439504:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9c000 x1862032664849280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439521:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664849280/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439528:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439530:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776044.439533:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776044.439542:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439546:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9df80 x1862032664849408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664849408/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439561:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439563:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439565:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776044.439568:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776044.439577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439581:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9ed80 x1862032664849536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664849536/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439593:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439594:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776044.439597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849664:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776044.439606:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9e680 x1862032664849664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439616:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664849664/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439624:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664849664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776044.439627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776044.439635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439639:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9d500 x1862032664849792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439645:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664849792/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776044.439663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776044.439671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776044.439676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776044/real 1775776044] req@ffff982980f9fb80 x1862032664849920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776044.439682:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664849920/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776044.439686:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776044.439688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776044.439689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664849920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776044.951138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776045.207243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776045.975124:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776046.231248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776046.999130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776047.255216:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776048.023221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776048.279239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776049.047150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776049.303279:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776049.559247:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559257:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776049.559261:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559272:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559284:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776049.559291:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776049.559337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664850048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559359:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776049.559364:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559367:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776049.559372:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559380:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776049.559389:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664850176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559396:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776049.559401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559405:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776049.559407:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559411:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776049.559418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664850304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559425:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559427:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776049.559429:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559432:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559438:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776049.559440:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664850048:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776049.559445:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559459:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776049.559472:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664850432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776049.559484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559489:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776049.559491:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559494:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776049.559500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664850560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559509:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776049.559510:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559513:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559517:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775776049.559523:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776049.559527:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776049.559540:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664850688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776049.559548:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776049.559550:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776049.559553:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776049.559557:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17817 00000100:00080000:1.0:1775776049.559559:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776049.559561:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe1180 x1862032664850048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776049.559566:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776049.559577:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776049.559595:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664850816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776049.559601:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664850048/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559624:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664850048:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776049.559648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776049.559673:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe1500 x1862032664850176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664850176/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559713:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776049.559734:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776049.559750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559757:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe1c00 x1862032664850304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559764:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664850304/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776049.559777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776049.559798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe0e00 x1862032664850432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664850432/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559817:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776049.559822:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776049.559834:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe0a80 x1862032664850560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559848:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664850560/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664850560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776049.559859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664850688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776049.559868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559876:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe1f80 x1862032664850688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559887:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664850688/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559894:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559900:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664850688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776049.559906:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664850816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776049.559922:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776049.559931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776049/real 1775776049] req@ffff982981fe3b80 x1862032664850816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776049.559941:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664850816/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776049.559953:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776049.559957:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776049.559960:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664850816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776050.071220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776050.327250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776051.095168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776051.351241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776052.119169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776052.375251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776053.143180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776053.399217:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776054.167183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776054.423295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776054.679236:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679243:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776054.679246:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679253:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679263:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679268:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776054.679300:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664850944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679327:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679329:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776054.679330:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679333:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679335:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679337:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776054.679351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664851072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679357:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679359:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776054.679360:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679362:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679366:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776054.679373:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664850944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776054.679378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664851200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776054.679388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679390:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679394:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776054.679405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664851328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776054.679414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679416:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679419:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679421:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776054.679427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776054.679431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664851456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679436:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776054.679439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679441:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679442:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679444:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679446:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776054.679450:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664851584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776054.679454:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776054.679456:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776054.679457:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776054.679459:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17822 00000100:00080000:0.0:1775776054.679460:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776054.679462:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776054.679464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776054.679468:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664851712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776054.679568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2c380 x1862032664850944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.679587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664850944/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.679596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.679600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.679603:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664850944:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776054.679613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776054.679661:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.679669:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2f480 x1862032664851072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.679676:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664851072/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.679684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.679687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.679690:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776054.679722:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776054.679742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.679750:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2fb80 x1862032664851200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.679758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664851200/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.679767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.679769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.679771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776054.679775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776054.679787:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.679793:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2d180 x1862032664851328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.679799:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664851328/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.679803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.679805:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.679806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776054.679810:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776054.679821:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.679826:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2e300 x1862032664851456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.679832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664851456/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.679836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.679838:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.680981:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776054.680987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664851584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776054.681009:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.681019:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2d500 x1862032664851584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.681026:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664851584/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.681031:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.681045:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.681048:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664851584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776054.681052:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664851712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776054.681078:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776054.681085:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776054/real 1775776054] req@ffff982974b2e680 x1862032664851712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776054.681094:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664851712/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776054.681101:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776054.681103:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776054.681106:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664851712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776055.191128:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776055.447214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776056.215166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776056.471244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776057.239182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776057.495230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776058.263160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776058.519261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776059.287164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776059.543236:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776059.799142:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799149:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776059.799152:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799160:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799169:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776059.799174:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776059.799206:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664851840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776059.799226:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799228:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776059.799229:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799232:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799234:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776059.799236:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799244:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776059.799251:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664851968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776059.799258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799260:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776059.799261:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799265:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799267:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776059.799267:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664851840:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776059.799270:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799275:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776059.799281:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664852096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776059.799286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799288:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776059.799288:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799291:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799293:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776059.799295:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776059.799307:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664852224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776059.799312:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776059.799316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799318:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799321:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776059.799323:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799326:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776059.799331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664852352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776059.799338:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776059.799340:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799343:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776059.799344:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799347:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776059.799351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799353:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776059.799359:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664852480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776059.799365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776059.799368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776059.799369:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776059.799371:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17827 00000100:00080000:0.0:1775776059.799373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776059.799376:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2df80 x1862032664851840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776059.799377:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776059.799382:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776059.799390:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664852608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776059.799405:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664851840/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799417:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799422:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799426:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664851840:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776059.799454:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776059.799478:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799490:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2ca80 x1862032664851968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799502:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664851968/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799511:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799518:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664851968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776059.799524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776059.799543:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799553:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2f100 x1862032664852096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799566:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664852096/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799580:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799584:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776059.799593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776059.799613:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2e680 x1862032664852224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664852224/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799641:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776059.799653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776059.799671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2d500 x1862032664852352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664852352/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776059.799712:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664852480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776059.799731:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799741:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2e300 x1862032664852480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664852480/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799764:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664852480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776059.799772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664852608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776059.799790:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776059.799800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776059/real 1775776059] req@ffff982974b2d180 x1862032664852608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776059.799811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664852608/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776059.799819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776059.799822:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776059.799826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664852608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776060.311166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776060.567257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776061.335227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776061.591222:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776062.359205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776062.615264:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776063.383206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776063.639250:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776064.407168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776064.663269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776064.919181:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776064.919190:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776064.919194:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776064.919201:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00080000:0.0:1775776064.919212:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919217:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776064.919249:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664852736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776064.919278:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776064.919281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776064.919283:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776064.919287:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00080000:0.0:1775776064.919291:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919294:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776064.919310:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776064.919319:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664852736:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776064.919326:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664852864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776064.919338:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776064.919342:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776064.919344:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776064.919348:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00080000:0.0:1775776064.919352:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919356:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776064.919360:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776064.919372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664852992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776064.919375:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776064.919385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776064.919388:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776064.919391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776064.919395:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00080000:0.0:1775776064.919400:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919405:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919406:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2fb80 x1862032664852736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776064.919410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776064.919428:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664853120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776064.919432:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664852736/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776064.919448:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776064.919452:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776064.919453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776064.919456:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776064.919460:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776064.919464:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00100000:3.0:1775776064.919466:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664852736:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776064.919471:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919477:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776064.919482:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776064.919488:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776064.919495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664853248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776064.919506:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776064.919510:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776064.919514:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776064.919516:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776064.919521:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00100000:3.0:1775776064.919524:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2f480 x1862032664852864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776064.919526:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919533:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919537:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664852864/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776064.919538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776064.919547:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919550:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776064.919555:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664853376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776064.919555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776064.919561:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852992:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776064.919573:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776064.919576:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776064.919579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776064.919579:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776064.919587:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17832 00000100:00100000:3.0:1775776064.919590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2c380 x1862032664852992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776064.919591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776064.919598:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664852992/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776064.919612:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776064.919616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919621:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776064.919624:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664853504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776064.919628:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664852992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776064.919640:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776064.919663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2d180 x1862032664853120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776064.919679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664853120/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776064.919684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776064.919688:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776064.919699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776064.919711:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919716:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2e300 x1862032664853248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776064.919722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664853248/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776064.919728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919729:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776064.919732:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776064.919734:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664853376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776064.919742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919747:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2d500 x1862032664853376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776064.919753:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664853376/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776064.919758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919759:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776064.919760:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664853376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776064.919763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664853504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776064.919771:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776064.919775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776064/real 1775776064] req@ffff982974b2e680 x1862032664853504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776064.919781:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664853504/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776064.919785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776064.919787:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776064.919789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664853504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776065.431172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776065.687234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776066.455156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776066.711236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776067.479247:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776067.735262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776068.503125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776068.759213:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776069.527188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776069.783233:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776070.039200:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776070.039210:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776070.039214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039225:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:2.0:1775776070.039236:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776070.039243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776070.039283:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664853632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776070.039313:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776070.039317:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776070.039320:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039324:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:2.0:1775776070.039328:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776070.039332:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776070.039342:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776070.039355:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664853760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776070.039362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664853632:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776070.039379:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776070.039384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776070.039386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039390:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:2.0:1775776070.039395:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776070.039399:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776070.039405:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776070.039416:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664853888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776070.039426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776070.039429:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776070.039439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776070.039441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039446:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:2.0:1775776070.039451:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776070.039456:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776070.039462:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776070.039463:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b5f80 x1862032664853632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776070.039477:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664854016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776070.039482:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664853632/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776070.039497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776070.039501:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776070.039503:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039512:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:3.0:1775776070.039513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776070.039517:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776070.039520:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776070.039524:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039528:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664853632:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775776070.039531:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776070.039545:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664854144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776070.039557:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776070.039558:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853760:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776070.039566:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776070.039568:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776070.039573:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:2.0:1775776070.039578:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776070.039582:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776070.039587:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776070.039592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776070.039604:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664854272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776070.039610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b4700 x1862032664853760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776070.039617:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776070.039622:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776070.039624:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776070.039625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664853760/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776070.039631:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17837 00000100:00080000:3.0:1775776070.039636:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776070.039639:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776070.039641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776070.039647:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039647:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776070.039656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853888:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775776070.039666:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776070.039671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039678:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b5500 x1862032664853888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776070.039685:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664854400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776070.039688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664853888/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776070.039710:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776070.039715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776070.039718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664853888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776070.039723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854016:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776070.039735:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039742:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b4a80 x1862032664854016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776070.039749:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664854016/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776070.039754:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776070.039755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776070.039757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776070.039788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854144:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776070.039801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b6d80 x1862032664854144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776070.039814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664854144/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776070.039818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776070.039820:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776070.039822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776070.039825:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664854272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776070.039835:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039843:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b5180 x1862032664854272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776070.039853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664854272/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776070.039860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776070.039863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776070.039866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664854272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776070.039871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664854400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776070.039888:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776070.039898:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776070/real 1775776070] req@ffff9829764b5c00 x1862032664854400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776070.039908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664854400/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776070.040025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776070.040034:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776070.040040:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664854400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776070.551121:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776070.807221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776071.575170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776071.831227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776072.599168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776072.855198:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776073.623156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776073.879260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776074.647190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776074.903211:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776075.159156:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776075.159165:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776075.159169:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159178:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159189:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776075.159195:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776075.159233:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664854528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776075.159274:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776075.159277:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776075.159279:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159284:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159288:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776075.159292:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776075.159300:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776075.159314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664854656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776075.159325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776075.159327:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664854528:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776075.159328:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776075.159332:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159338:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159341:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776075.159344:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776075.159347:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776075.159357:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664854784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776075.159363:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776075.159366:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776075.159368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159372:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159389:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776075.159393:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776075.159399:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775776075.159401:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776075.159427:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664854912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776075.159435:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b6680 x1862032664854528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776075.159442:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776075.159452:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776075.159455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159461:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159465:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776075.159465:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664854528/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776075.159473:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776075.159483:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776075.159483:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159490:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159494:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664854528:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776075.159499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664855040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776075.159510:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776075.159513:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776075.159515:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159519:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159523:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776075.159524:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854656:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776075.159531:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776075.159537:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776075.159547:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664855168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776075.159555:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776075.159555:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776075.159560:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776075.159561:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776075.159565:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17842 00000100:00080000:2.0:1775776075.159569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776075.159574:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159575:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b4000 x1862032664854656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776075.159577:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776075.159587:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664855296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776075.159603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664854656/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159613:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776075.159626:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776075.159642:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b6a00 x1862032664854784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776075.159661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664854784/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776075.159704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776075.159721:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b7100 x1862032664854912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776075.159740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664854912/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159749:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159753:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664854912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776075.159758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776075.159776:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b7b80 x1862032664855040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776075.159797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664855040/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159804:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159807:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159810:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776075.159828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664855168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776075.159845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159853:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b4e00 x1862032664855168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776075.159864:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664855168/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664855168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776075.159881:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664855296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776075.159899:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776075.159909:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776075/real 1775776075] req@ffff9829764b6680 x1862032664855296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776075.159920:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664855296/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776075.159928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776075.159931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776075.159934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664855296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776075.671191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776075.927219:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776076.695188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776076.951228:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776077.719187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776077.975254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776078.743184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776078.999192:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776079.767174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776080.023222:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776080.279201:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776080.279210:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776080.279214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776080.279223:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279233:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776080.279239:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776080.279276:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664855424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776080.279305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776080.279309:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776080.279311:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776080.279315:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279319:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776080.279325:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776080.279334:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776080.279344:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664855424:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776080.279348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664855552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776080.279361:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776080.279365:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776080.279366:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776080.279371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279375:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776080.279379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776080.279384:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776080.279395:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664855680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776080.279400:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776080.279409:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776080.279414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776080.279416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776080.279421:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279425:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776080.279428:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9f800 x1862032664855424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776080.279432:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776080.279442:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776080.279448:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664855424/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279456:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775776080.279456:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664855808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776080.279463:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776080.279470:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664855424:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776080.279476:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776080.279480:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776080.279483:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776080.279487:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279491:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775776080.279493:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855552:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776080.279498:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776080.279510:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775776080.279513:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776080.279523:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9c700 x1862032664855552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776080.279528:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664855936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776080.279535:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664855552/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776080.279547:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776080.279551:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776080.279554:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775776080.279556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855552:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776080.279557:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776080.279562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776080.279566:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279572:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776080.279577:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775776080.279578:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776080.279585:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776080.279608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9c380 x1862032664855680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776080.279614:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664856064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776080.279621:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664855680/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776080.279632:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776080.279633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776080.279638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776080.279640:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775776080.279642:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776080.279644:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776080.279649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855808:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776080.279651:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17847 00000100:00080000:3.0:1775776080.279657:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776080.279661:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:0.0:1775776080.279668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776080.279674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9fb80 x1862032664855808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776080.279681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664855808/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279685:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776080.279687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776080.279689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776080.279700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776080.279709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776080.279714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9d500 x1862032664855936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776080.279721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664855936/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776080.279726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776080.279728:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664855936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776080.279731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776080.279740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776080.279744:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9e680 x1862032664856064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776080.279751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664856064/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776080.279767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776080.279769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856064:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775776080.279775:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776080.279806:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664856192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776080.279853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776080.279874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776080.279882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776080/real 1775776080] req@ffff982980f9ed80 x1862032664856192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776080.279889:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664856192/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776080.279893:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776080.279895:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776080.279897:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776080.791182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776081.047250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776081.815176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776082.071225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776082.839206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776083.095260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776083.863208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776084.119246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776084.887217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776085.143184:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776085.399184:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776085.399193:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776085.399197:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399209:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399223:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399230:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776085.399274:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664856320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399305:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776085.399308:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776085.399310:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399313:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399316:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399328:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776085.399340:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664856448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399352:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776085.399355:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664856320:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776085.399355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776085.399358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399364:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399369:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399371:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776085.399378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664856576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399384:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776085.399387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776085.399388:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399390:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399392:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399394:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776085.399403:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664856704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776085.399411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776085.399413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399415:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399418:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775776085.399427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776085.399430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664856832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776085.399437:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776085.399439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399441:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399447:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399449:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776085.399457:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664856960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776085.399462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775776085.399463:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9df80 x1862032664856320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776085.399465:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776085.399468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776085.399474:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17852 00000100:00080000:3.0:1775776085.399479:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776085.399483:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776085.399486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776085.399492:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664857088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775776085.399496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664856320/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399541:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664856320:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776085.399565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776085.399581:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9c000 x1862032664856448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399602:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664856448/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399607:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776085.399615:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776085.399624:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9d180 x1862032664856576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664856576/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399647:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776085.399654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776085.399664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9f480 x1862032664856704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399676:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664856704/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399688:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776085.399691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776085.399701:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9ea00 x1862032664856832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664856832/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664856832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776085.399739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776085.399757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9dc00 x1862032664856960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664856960/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399781:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664856960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776085.399784:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776085.399796:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776085.399804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776085/real 1775776085] req@ffff982980f9d880 x1862032664857088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776085.399814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664857088/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776085.399822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776085.399825:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776085.399828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776085.911176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776086.167272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776086.935209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776087.191297:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776087.959163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776088.215234:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776088.983180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776089.239266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776090.007232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776090.263250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776090.519145:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519151:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776090.519155:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776090.519162:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519171:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519176:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776090.519206:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664857216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519225:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776090.519228:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776090.519231:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519232:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519234:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519241:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776090.519248:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664857344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519255:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519257:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776090.519258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776090.519262:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664857216:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776090.519264:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519268:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519271:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519274:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776090.519282:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664857472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519288:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519290:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776090.519291:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776090.519293:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519295:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519297:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519299:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776090.519305:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664857600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519311:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519313:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776090.519314:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775776090.519318:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776090.519320:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519323:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519325:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519328:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776090.519335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664857728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519341:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776090.519344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776090.519346:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775776090.519347:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b5c00 x1862032664857216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776090.519350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00080000:2.0:1775776090.519354:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519359:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776090.519374:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664857216/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776090.519374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664857856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776090.519402:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776090.519404:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776090.519405:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776090.519407:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776090.519420:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776090.519429:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17858 00000100:00100000:0.0:1775776090.519430:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664857216:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776090.519434:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776090.519440:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776090.519444:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776090.519458:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775776090.519459:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664857984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775776090.519485:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519495:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b5180 x1862032664857344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519503:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664857344/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519508:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519510:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519512:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776090.519516:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776090.519526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b6d80 x1862032664857472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519538:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664857472/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519543:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519546:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776090.519549:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776090.519559:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519564:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b4a80 x1862032664857600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519571:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664857600/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519578:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776090.519581:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:0.0:1775776090.519590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b5500 x1862032664857728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519601:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664857728/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519608:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664857728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776090.519611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776090.519620:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519624:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b4700 x1862032664857856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664857856/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519635:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519637:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776090.519654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:0.0:1775776090.519672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776090.519682:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776090/real 1775776090] req@ffff9829764b5f80 x1862032664857984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776090.519692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664857984/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776090.519699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776090.519702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776090.519705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664857984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776091.031175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776091.287195:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776092.055180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776092.311208:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776093.079170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776093.335284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776094.103196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776094.359182:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776095.127182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776095.383267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776095.639199:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639209:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776095.639214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639228:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639251:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776095.639299:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664858112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639349:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776095.639351:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639356:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639360:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639366:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639377:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776095.639389:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664858112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776095.639392:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664858240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776095.639415:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639419:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639423:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639425:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639430:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776095.639440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664858368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776095.639455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639457:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639460:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639463:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639466:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776095.639476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664858496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639481:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639485:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776095.639487:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639490:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639492:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639496:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639499:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776095.639505:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776095.639508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664858624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639527:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639528:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776095.639529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639533:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639535:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639539:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639543:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776095.639554:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664858752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776095.639560:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776095.639563:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776095.639564:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776095.639566:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17863 00000100:00080000:0.0:1775776095.639570:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776095.639573:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776095.639577:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776095.639577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776095.639592:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664858880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776095.639618:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776095.639641:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776095.639658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664858752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776095.639671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664858880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776096.151165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776096.407270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776097.175223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776097.431245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776098.199158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776098.455258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775776098.711326:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775776098.711371:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775776098.711392:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713111:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713125:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713134:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713144:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713153:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776098.713165:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775776098.713208:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2d500 x1862032664858624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713243:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664858624/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713253:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713258:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713261:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776098.713283:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2e300 x1862032664858752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713296:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664858752/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713315:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713319:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713322:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664858752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776098.713328:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2d180 x1862032664858880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713339:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664858880/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713347:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713350:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713352:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664858880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776098.713358:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2ca80 x1862032664858240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713369:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664858240/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713386:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713390:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713393:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776098.713399:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2df80 x1862032664858368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713410:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664858368/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713418:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713421:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713424:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776098.713430:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2e680 x1862032664858496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713439:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664858496/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713446:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713449:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713452:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664858496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776098.713459:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776095/real 1775776098] req@ffff982974b2f100 x1862032664858112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776098.713469:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664858112/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776098.713475:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776098.713479:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776098.713481:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664858112:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776099.223141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776099.479282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776100.247199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776100.503225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776100.759274:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759284:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776100.759287:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759297:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759309:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759314:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776100.759349:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664859008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776100.759383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759387:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759394:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776100.759417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664859136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776100.759430:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759435:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759439:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759442:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776100.759461:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664859008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776100.759461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664859264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759486:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759495:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776100.759497:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759501:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759506:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759510:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776100.759530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664859392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759541:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759544:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776100.759547:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759551:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759555:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759559:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759565:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776100.759577:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664859520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759588:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775776100.759590:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776100.759598:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776100.759602:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776100.759607:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759612:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759617:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759625:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776100.759637:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664859648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776100.759648:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776100.759652:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776100.759657:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776100.759662:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9ce00 x1862032664859008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776100.759669:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17868 00000100:00080000:3.0:1775776100.759679:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776100.759690:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776100.759697:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776100.759707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664859008/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776100.759715:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664859776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776100.759733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.759745:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.759751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664859008:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776100.759766:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776100.759802:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.759816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9e300 x1862032664859136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.759828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664859136/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.759836:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.759839:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.759843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776100.759878:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776100.759900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.759911:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9f100 x1862032664859264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.759923:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664859264/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.759937:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.759941:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.759944:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776100.759950:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859392:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776100.759974:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.759986:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9d880 x1862032664859392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.759998:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664859392/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.760007:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.760010:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.760013:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859392:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776100.760020:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776100.760041:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.760052:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9dc00 x1862032664859520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.760063:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664859520/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.760132:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.760136:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.760140:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664859520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776100.760148:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664859648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776100.760171:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.760186:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9ea00 x1862032664859648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.760199:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664859648/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.760208:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.760211:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.760216:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664859648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776100.760223:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664859776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776100.760260:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776100.760273:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776100/real 1775776100] req@ffff982980f9f480 x1862032664859776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776100.760291:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664859776/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776100.760300:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776100.760303:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776100.760305:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664859776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776101.271155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776101.527262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776102.295173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776102.551260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776103.319202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776103.575223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776104.343226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776104.599335:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776105.367147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776105.623255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776105.879197:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776105.879206:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776105.879210:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879219:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00080000:0.0:1775776105.879232:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879238:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776105.879274:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664859904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776105.879302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776105.879306:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776105.879308:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879312:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00080000:0.0:1775776105.879316:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879320:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776105.879342:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664860032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776105.879345:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664859904:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776105.879360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776105.879363:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776105.879365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879370:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00080000:0.0:1775776105.879375:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879384:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776105.879396:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664860160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776105.879406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775776105.879408:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776105.879412:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776105.879414:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879419:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00080000:0.0:1775776105.879423:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879427:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776105.879439:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2c380 x1862032664859904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776105.879445:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664860288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776105.879457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776105.879460:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664859904/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776105.879461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776105.879464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879471:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00080000:1.0:1775776105.879474:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776105.879475:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776105.879482:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776105.879483:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776105.879487:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664859904:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776105.879499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664860416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776105.879505:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776105.879510:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860032:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776105.879516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776105.879518:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776105.879524:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17873 00000800:00000100:1.0:1775776105.879528:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776105.879530:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879534:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879538:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776105.879539:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2f480 x1862032664860032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879549:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664860032/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776105.879552:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664860544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776105.879561:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776105.879567:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776105.879568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776105.879573:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776105.879575:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776105.879575:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860032:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776105.879583:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17873 00000100:00100000:1.0:1775776105.879586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860160:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776105.879587:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776105.879593:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776105.879596:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776105.879606:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776105.879607:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664860672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776105.879616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2fb80 x1862032664860160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664860160/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776105.879632:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776105.879635:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776105.879636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776105.879639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860288:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776105.879650:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776105.879656:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2f100 x1862032664860288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879662:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664860288/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776105.879666:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776105.879667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776105.879669:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860288:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776105.879672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776105.879680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776105.879684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2e680 x1862032664860416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664860416/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776105.879698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776105.879700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776105.879702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776105.879710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664860544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776105.879719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776105.879723:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2df80 x1862032664860544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664860544/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776105.879733:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776105.879735:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776105.879746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664860544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776105.879750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664860672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776105.879759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776105.879765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776105/real 1775776105] req@ffff982974b2ca80 x1862032664860672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776105.879771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664860672/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776105.879776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776105.879777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776105.879780:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664860672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776106.391178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776106.647243:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776107.415163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776107.671269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776108.439184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776108.695281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776109.463182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776109.719218:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775776110.487207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776110.743237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776110.999239:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999259:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776110.999266:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776110.999279:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999294:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776110.999303:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776110.999348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664860800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776110.999381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776110.999387:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776110.999392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999396:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776110.999400:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776110.999411:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776110.999425:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664860928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776110.999436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776110.999443:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776110.999444:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664860800:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776110.999449:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999455:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776110.999460:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776110.999473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776110.999488:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664861056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776110.999504:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999507:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776110.999509:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776110.999513:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999517:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776110.999521:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776110.999526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776110.999528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776110.999546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664861184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776110.999555:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999560:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776110.999562:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776110.999563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2d180 x1862032664860800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776110.999570:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999577:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776110.999583:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776110.999588:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776110.999588:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664860800/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776110.999605:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664861312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776110.999606:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664860800:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776110.999623:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776110.999627:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776110.999629:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776110.999633:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999638:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776110.999640:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860928:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776110.999645:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776110.999652:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776110.999666:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664861440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776110.999672:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776110.999683:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776110.999684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2e300 x1862032664860928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776110.999693:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776110.999697:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776110.999701:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664860928/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776110.999703:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17878 00000100:00080000:0.0:1775776110.999711:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776110.999711:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776110.999717:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776110.999723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000008:00100000:0.0:1775776110.999725:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776110.999726:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664860928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776110.999733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776110.999742:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664861568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776110.999750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776110.999759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2d500 x1862032664861056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776110.999767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664861056/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776110.999775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776110.999781:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861184:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776110.999794:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776110.999800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2ca80 x1862032664861184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776110.999808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664861184/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776110.999815:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861184:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776110.999820:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861312:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776110.999829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776110.999840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2df80 x1862032664861312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776110.999851:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664861312/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999856:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776110.999858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776110.999870:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664861440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776110.999887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776110.999897:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2e680 x1862032664861440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776110.999908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664861440/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776110.999918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664861440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776110.999923:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664861568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776110.999936:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776110.999942:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776110/real 1775776110] req@ffff982974b2f100 x1862032664861568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776110.999949:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664861568/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776110.999957:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776110.999960:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776110.999961:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664861568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776111.511199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776111.767192:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776112.535177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776112.791200:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776113.559187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776113.815227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776114.583174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776114.839243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776115.607184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776115.863199:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776116.119189:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119197:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776116.119201:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119209:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119220:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119225:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776116.119260:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664861696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119282:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119285:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776116.119286:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119289:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119291:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119294:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119302:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776116.119311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664861824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119318:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119321:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776116.119322:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119325:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119327:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119332:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776116.119337:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664861696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776116.119341:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664861952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119348:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776116.119356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119359:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119361:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119363:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776116.119374:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664862080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776116.119384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119386:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119388:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119390:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119393:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776116.119399:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664862208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119407:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776116.119408:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119412:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119414:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119417:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776116.119420:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776116.119427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664862336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776116.119433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776116.119436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776116.119438:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776116.119441:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17883 00000100:00080000:2.0:1775776116.119444:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776116.119446:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776116.119449:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776116.119456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664862464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776116.119465:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b5f80 x1862032664861696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119485:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664861696/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119497:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119501:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119505:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664861696:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776116.119513:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776116.119538:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b4700 x1862032664861824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119565:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664861824/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119573:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119577:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119581:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776116.119607:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776116.119631:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b5500 x1862032664861952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119655:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664861952/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119666:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664861952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776116.119677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862080:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776116.119698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b4a80 x1862032664862080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664862080/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862080:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776116.119744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776116.119766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b6d80 x1862032664862208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119790:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664862208/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776116.119811:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664862336:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776116.119830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119841:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b5180 x1862032664862336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664862336/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119865:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119867:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664862336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776116.119872:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664862464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776116.119891:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776116.119901:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776116/real 1775776116] req@ffff9829764b5c00 x1862032664862464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776116.119920:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664862464/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776116.119928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776116.119931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776116.119934:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664862464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776116.631193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776116.887250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776117.655176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776117.911259:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776118.679184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776118.935197:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776119.703151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776119.959214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776120.727147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776120.983239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776121.239189:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776121.239203:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776121.239212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239224:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239230:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776121.239263:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664862592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776121.239287:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776121.239293:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776121.239297:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239299:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776121.239311:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776121.239322:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664862720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776121.239332:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776121.239339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776121.239339:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664862592:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776121.239344:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776121.239356:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776121.239365:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664862848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776121.239375:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239378:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776121.239379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776121.239382:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239394:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239398:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776121.239403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776121.239417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664862976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776121.239418:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776121.239428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776121.239434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776121.239438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239442:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239446:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776121.239451:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776121.239456:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2fb80 x1862032664862592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776121.239463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664863104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776121.239479:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776121.239484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776121.239487:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664862592/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776121.239490:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239496:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239503:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776121.239506:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776121.239507:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776121.239514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776121.239520:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664863232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776121.239521:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664862592:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776121.239540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776121.239544:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776121.239546:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776121.239548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862720:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776121.239553:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17888 00000100:00080000:0.0:1775776121.239558:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776121.239561:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776121.239566:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776121.239577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776121.239580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664863360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776121.239593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2f480 x1862032664862720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239606:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664862720/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239626:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776121.239632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776121.239649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776121.239659:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2c380 x1862032664862848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239670:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664862848/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239677:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239680:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776121.239687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862976:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776121.239704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776121.239713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2f100 x1862032664862976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239723:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664862976/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239737:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664862976:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776121.239742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776121.239757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776121.239766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2e680 x1862032664863104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664863104/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239782:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776121.239794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664863232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776121.239810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776121.239820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2df80 x1862032664863232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239831:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664863232/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239840:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239843:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664863232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776121.239859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664863360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776121.239893:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776121.239904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776121/real 1775776121] req@ffff982974b2ca80 x1862032664863360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776121.239914:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664863360/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776121.239922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776121.239924:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776121.239927:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664863360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776121.751163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776122.007240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776122.775196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776123.031235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776123.799164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776124.055237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776124.823156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776125.079231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776125.847223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776126.103261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776126.359236:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776126.359251:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359260:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359271:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359277:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776126.359309:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664863488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776126.359340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359345:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776126.359355:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776126.359365:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664863616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359376:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776126.359382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359392:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359396:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359400:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776126.359407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776126.359413:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664863488:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776126.359422:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664863744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359433:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776126.359439:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359443:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359447:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359451:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776126.359457:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776126.359470:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664863872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359479:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776126.359484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359498:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359502:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776126.359506:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776126.359513:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776126.359517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664864000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776126.359531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359536:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00080000:3.0:1775776126.359543:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359549:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776126.359556:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776126.359563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9d180 x1862032664863488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776126.359570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664864128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776126.359605:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776126.359609:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776126.359610:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776126.359616:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17893 00000100:00100000:1.0:1775776126.359617:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664863488/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776126.359624:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776126.359633:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776126.359642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775776126.359642:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776126.359651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.359661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664863488:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776126.359661:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664864256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776126.359685:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776126.359717:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.359733:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9c000 x1862032664863616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.359756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664863616/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.359764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.359768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.359771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776126.359817:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776126.359840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.359852:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9df80 x1862032664863744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.359864:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664863744/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.359872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.359876:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.359879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776126.359886:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863872:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776126.359908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.359920:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9ed80 x1862032664863872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.359932:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664863872/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.359946:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.359950:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.359953:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664863872:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776126.359959:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776126.359982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.359992:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9e680 x1862032664864000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.360003:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664864000/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.360011:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.360015:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.360018:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776126.360024:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664864128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776126.360045:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.360055:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9d500 x1862032664864128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.360099:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664864128/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.360108:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.360111:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.360115:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664864128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776126.360121:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664864256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776126.360143:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776126.360154:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776126/real 1775776126] req@ffff982980f9fb80 x1862032664864256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776126.360166:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664864256/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776126.360174:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776126.360177:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776126.360180:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664864256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776126.871156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776127.127223:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776127.895181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776128.151291:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776128.919191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776129.175190:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776129.943197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776130.199371:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776130.967183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776131.223179:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776131.479163:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479170:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776131.479174:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479182:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479192:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479196:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776131.479227:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664864384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776131.479248:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479251:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776131.479252:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479254:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479256:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479258:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479265:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776131.479272:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664864512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776131.479278:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479280:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776131.479281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479285:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479287:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479289:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776131.479295:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664864640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776131.479303:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664864384:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776131.479306:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479311:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776131.479313:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479315:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479317:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479319:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479322:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776131.479330:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664864768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776131.479335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776131.479339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479341:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479343:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479345:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479347:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776131.479352:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664864896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776131.479358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479360:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776131.479361:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479363:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479368:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479372:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776131.479373:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776131.479380:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664865024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776131.479386:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776131.479388:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776131.479389:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776131.479392:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17898 00000100:00080000:3.0:1775776131.479394:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776131.479396:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776131.479398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776131.479406:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664865152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776131.479413:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9c380 x1862032664864384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479431:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664864384/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479441:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479445:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479449:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664864384:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776131.479456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776131.479480:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9c700 x1862032664864512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479504:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664864512/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479511:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479517:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776131.479541:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776131.479568:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479578:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9f800 x1862032664864640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479585:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664864640/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479590:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479592:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479594:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776131.479597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864768:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776131.479612:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9fb80 x1862032664864768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479637:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664864768/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479645:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479648:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864768:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776131.479656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776131.479675:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9d500 x1862032664864896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664864896/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479723:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479727:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664864896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776131.479731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776131.479744:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9e680 x1862032664865024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664865024/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479765:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479769:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776131.479772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776131.479780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776131.479785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776131/real 1775776131] req@ffff982980f9ed80 x1862032664865152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776131.479790:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664865152/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776131.479795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776131.479796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776131.479797:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776131.991179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776132.247259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776133.015221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776133.271272:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776134.039196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776134.295254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776135.063215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776135.319227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776136.087201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776136.343273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776136.599159:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599166:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776136.599169:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599177:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599187:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599193:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776136.599229:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664865280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776136.599258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599261:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776136.599263:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599266:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599269:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599274:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599283:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776136.599297:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664865408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776136.599307:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776136.599312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599316:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599320:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599324:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599329:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776136.599344:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664865536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776136.599346:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664865280:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776136.599365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776136.599368:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599372:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599375:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599378:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776136.599391:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664865664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776136.599397:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599400:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776136.599401:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599404:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599407:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599410:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599412:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776136.599417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664865792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776136.599423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776136.599426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599428:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599430:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599432:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776136.599441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664865920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776136.599446:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776136.599455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776136.599457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776136.599458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776136.599460:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17904 00000100:00080000:3.0:1775776136.599463:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776136.599465:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776136.599469:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776136.599479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664866048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776136.599505:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9df80 x1862032664865280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599526:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664865280/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599538:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599542:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599546:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664865280:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776136.599554:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776136.599576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599587:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9c000 x1862032664865408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599599:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664865408/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599626:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599628:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776136.599651:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776136.599665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599672:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9d180 x1862032664865536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599679:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664865536/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599684:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776136.599694:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865664:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776136.599704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9f480 x1862032664865664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664865664/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865664:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776136.599727:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865792:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776136.599737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599742:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9ea00 x1862032664865792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599748:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664865792/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664865792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776136.599759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776136.599768:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599774:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9dc00 x1862032664865920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664865920/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599787:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664865920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776136.599791:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776136.599801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776136.599806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776136/real 1775776136] req@ffff982980f9d880 x1862032664866048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776136.599813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664866048/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776136.599817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776136.599819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776136.599820:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776137.111198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776137.367232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776138.135226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776138.391361:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776139.159224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776139.415295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776140.183172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776140.439261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776141.207182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776141.463275:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776141.719207:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719219:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776141.719224:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719235:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719257:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719265:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776141.719322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664866176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776141.719356:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719360:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776141.719362:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719367:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719372:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719377:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719387:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776141.719400:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664866304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776141.719411:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664866176:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776141.719411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719415:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776141.719417:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719424:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719426:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719429:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719432:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776141.719441:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664866432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776141.719447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776141.719451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719453:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719455:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719457:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719461:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776141.719468:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664866560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776141.719478:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719481:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776141.719482:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719484:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719487:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719489:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719491:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776141.719498:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776141.719498:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664866688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776141.719505:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719508:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776141.719510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719514:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00080000:3.0:1775776141.719527:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719532:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719536:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776141.719548:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664866816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776141.719555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9f100 x1862032664866176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776141.719560:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776141.719572:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776141.719574:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776141.719579:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17909 00000100:00100000:1.0:1775776141.719583:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664866176/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776141.719584:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776141.719591:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776141.719598:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776141.719603:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775776141.719610:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664866944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776141.719611:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664866176:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776141.719653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776141.719681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9e300 x1862032664866304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664866304/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776141.719728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776141.719746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9ce00 x1862032664866432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664866432/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719782:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776141.719788:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866560:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776141.719801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9d880 x1862032664866560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664866560/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866560:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776141.719830:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776141.719840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9dc00 x1862032664866688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664866688/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664866688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776141.719883:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776141.719902:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719912:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9ea00 x1862032664866816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664866816/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719924:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719927:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776141.719931:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776141.719943:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776141.719948:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776141/real 1775776141] req@ffff982980f9f480 x1862032664866944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776141.719954:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664866944/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776141.719959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776141.719961:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776141.719963:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664866944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776142.231179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776142.487399:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776143.255203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776143.511247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776144.279179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776144.535263:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776145.303191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776145.559218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776146.327135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776146.583235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776146.839229:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839240:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776146.839245:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839263:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839279:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776146.839286:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776146.839330:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664867072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776146.839360:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776146.839367:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839374:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776146.839379:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839391:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776146.839407:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664867200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776146.839417:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664867072:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776146.839420:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839453:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776146.839455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839459:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839464:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776146.839468:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776146.839487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664867328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776146.839496:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839500:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776146.839502:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839506:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839510:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776146.839516:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776146.839517:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839524:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776146.839537:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664867456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776146.839545:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839549:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776146.839551:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839555:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839559:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776146.839565:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9d180 x1862032664867072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776146.839565:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839573:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776146.839587:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664867584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776146.839595:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664867072/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776146.839597:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839605:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776146.839609:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839616:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:1.0:1775776146.839618:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776146.839620:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776146.839627:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776146.839627:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839635:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776146.839636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664867072:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776146.839646:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664867712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776146.839654:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776146.839658:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776146.839659:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776146.839662:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17914 00000100:00080000:3.0:1775776146.839666:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776146.839672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867200:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776146.839672:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776146.839680:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776146.839691:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664867840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776146.839707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.839721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9c000 x1862032664867200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.839735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664867200/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.839744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.839748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.839751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776146.839757:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776146.839778:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.839789:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9df80 x1862032664867328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.839801:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664867328/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.839809:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.839813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.839816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776146.839822:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867456:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776146.839844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.839854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9ed80 x1862032664867456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.839866:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664867456/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.839874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.839877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.839880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867456:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776146.839886:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776146.839906:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.839916:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9e680 x1862032664867584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.839928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664867584/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.839936:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.839939:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.839942:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664867584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776146.839948:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664867712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776146.839968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.839978:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9d500 x1862032664867712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.839989:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664867712/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.839997:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.840000:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.840003:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664867712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776146.840022:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664867840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776146.840043:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776146.840053:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776146/real 1775776146] req@ffff982980f9fb80 x1862032664867840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776146.840101:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664867840/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776146.840109:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776146.840112:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776146.840115:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664867840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776147.351222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776147.607251:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776148.375136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776148.631699:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776149.399184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776149.655337:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776150.423221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776150.679234:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776151.447166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776151.703265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776151.959244:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959252:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776151.959257:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959267:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:3.0:1775776151.959279:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776151.959285:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776151.959335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664867968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776151.959385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776151.959391:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959396:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:3.0:1775776151.959400:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776151.959404:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959415:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776151.959427:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664868096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776151.959435:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664867968:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776151.959442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959446:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776151.959447:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:3.0:1775776151.959457:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776151.959461:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959466:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776151.959479:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664868224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776151.959488:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959491:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776151.959493:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959497:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:3.0:1775776151.959501:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775776151.959505:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776151.959506:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959513:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776151.959527:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664868352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776151.959536:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959540:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776151.959542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959546:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:3.0:1775776151.959550:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776151.959555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9f800 x1862032664867968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776151.959555:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959560:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776151.959574:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664868480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776151.959583:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664867968/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776151.959585:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959591:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776151.959595:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959604:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00080000:1.0:1775776151.959606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776151.959607:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776151.959615:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776151.959615:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959623:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776151.959625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664867968:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776151.959637:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664868608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776151.959647:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776151.959650:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776151.959651:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776151.959654:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17919 00000100:00100000:1.0:1775776151.959657:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868096:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776151.959658:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776151.959664:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776151.959669:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776151.959679:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664868736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776151.959685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959699:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9c700 x1862032664868096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664868096/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959717:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776151.959727:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776151.959738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9c380 x1862032664868224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664868224/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959763:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959765:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776151.959770:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868352:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776151.959781:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959786:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9fb80 x1862032664868352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959793:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664868352/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868352:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776151.959806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776151.959815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9d500 x1862032664868480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664868480/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776151.959837:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664868608:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776151.959846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959851:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9e680 x1862032664868608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664868608/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959862:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959865:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664868608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776151.959878:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664868736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776151.959887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776151.959893:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776151/real 1775776151] req@ffff982980f9ed80 x1862032664868736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776151.959900:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664868736/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776151.959904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776151.959906:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776151.959908:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664868736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776152.471201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776152.727252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776153.495231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776153.751264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776154.519245:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776154.775255:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776155.543198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776155.799252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776156.567217:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776156.823272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776157.079207:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079215:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776157.079220:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079228:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079238:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079242:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776157.079290:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664868864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776157.079316:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776157.079319:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079323:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079325:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776157.079338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776157.079352:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664868992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776157.079362:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079365:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776157.079367:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079375:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079378:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079381:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776157.079386:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776157.079393:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664868864:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776157.079399:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664869120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776157.079418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776157.079421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079427:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079428:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776157.079432:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776157.079442:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664869248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776157.079449:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776157.079455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079458:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079478:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079484:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776157.079489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776157.079500:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776157.079504:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664869376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776157.079512:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079516:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776157.079517:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079520:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079526:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079529:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776157.079534:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776157.079544:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664869504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776157.079552:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2d500 x1862032664868864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776157.079556:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776157.079571:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776157.079576:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776157.079584:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17924 00000100:00080000:0.0:1775776157.079589:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776157.079591:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664868864/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776157.079598:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776157.079610:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664869632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776157.079631:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.079639:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.079645:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664868864:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776157.079656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776157.079685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2e300 x1862032664868992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.079711:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664868992/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.079716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.079718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.079720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664868992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776157.079743:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776157.079758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2d180 x1862032664869120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.079794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664869120/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.079803:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.079806:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.079809:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776157.079816:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869248:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776157.079837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079848:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2ca80 x1862032664869248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.079858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664869248/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.079871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.079874:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.079876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869248:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776157.079887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776157.079905:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079913:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2df80 x1862032664869376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.079933:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664869376/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.079948:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.079952:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.079955:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776157.079961:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664869504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776157.079979:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.079986:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2e680 x1862032664869504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.079993:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664869504/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.080005:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.080012:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.080016:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664869504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776157.080022:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664869632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776157.080042:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776157.080052:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776157/real 1775776157] req@ffff982974b2f100 x1862032664869632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776157.080063:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664869632/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776157.080102:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776157.080104:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776157.080107:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664869632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776157.591233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776157.847277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776158.615189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776158.871342:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776159.639174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776159.895264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776160.663181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776160.919283:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776161.687215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776161.943283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776162.199170:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199177:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776162.199180:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199189:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199206:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199210:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776162.199243:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664869760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776162.199272:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199276:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776162.199278:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199282:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199286:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199290:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199299:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776162.199314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664869888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776162.199330:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664869760:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776162.199333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199339:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776162.199340:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199344:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199348:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199352:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199357:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776162.199370:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664870016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776162.199377:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199380:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776162.199382:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199385:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199403:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199407:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776162.199424:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664870144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776162.199430:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776162.199434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199438:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00100000:1.0:1775776162.199441:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869888:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776162.199443:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199448:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199453:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776162.199461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664870272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776162.199468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199470:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776162.199471:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776162.199477:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870016:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776162.199477:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199479:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199483:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776162.199497:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664870400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776162.199499:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870144:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776162.199510:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776162.199512:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776162.199514:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776162.199517:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17929 00000100:00080000:3.0:1775776162.199518:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776162.199521:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776162.199523:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776162.199530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664870528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776162.199539:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776162.199565:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664870400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776162.199587:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664870528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776162.711181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776162.967267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776163.735171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776163.991249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776164.759182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776165.015244:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775776165.271380:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775776165.271430:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775776165.271453:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776165.271488:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776165.271497:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776165.271517:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776165.271529:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776165.271536:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776165.271542:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9c000 x1862032664869888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775776165.271548:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776165.271573:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664869888/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271588:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271591:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664869888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776165.271598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9d180 x1862032664870016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664870016/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271615:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271618:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776165.271626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9f480 x1862032664870144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271637:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664870144/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870144:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776165.271655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9df80 x1862032664869760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664869760/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271674:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271677:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271680:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664869760:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776165.271778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9d880 x1862032664870528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271790:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664870528/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664870528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776165.271810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9ea00 x1862032664870272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664870272/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776165.271841:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776162/real 1775776165] req@ffff982980f9dc00 x1862032664870400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776165.271852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664870400/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776165.271858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776165.271862:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776165.271864:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664870400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776165.783140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776166.039227:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776166.807165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776167.063245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776167.319210:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319217:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776167.319221:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319228:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00080000:0.0:1775776167.319237:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776167.319242:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776167.319272:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664870656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776167.319297:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319301:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776167.319303:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319309:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00080000:0.0:1775776167.319313:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776167.319318:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776167.319322:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664870656:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776167.319329:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776167.319346:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664870784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776167.319357:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319360:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776167.319361:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319366:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00080000:0.0:1775776167.319370:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776167.319374:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776167.319377:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776167.319382:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776167.319396:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664870912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776167.319400:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2c380 x1862032664870656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776167.319415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319419:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776167.319421:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00100000:3.0:1775776167.319428:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664870656/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776167.319431:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776167.319439:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776167.319444:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776167.319446:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776167.319452:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.319456:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664870656:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776167.319461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664871040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776167.319470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319473:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776167.319475:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319478:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00080000:0.0:1775776167.319482:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776167.319484:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870784:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776167.319488:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776167.319495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776167.319507:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664871168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776167.319516:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776167.319517:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319522:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776167.319524:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319528:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00100000:3.0:1775776167.319532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2f480 x1862032664870784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776167.319533:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776167.319541:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776167.319549:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776167.319554:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664870784/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776167.319562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664871296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776167.319564:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.319569:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.319572:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776167.319579:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870912:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776167.319591:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776167.319594:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776167.319595:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776167.319600:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17934 00000100:00080000:0.0:1775776167.319604:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776167.319608:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776167.319609:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776167.319617:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776167.319626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2fb80 x1862032664870912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776167.319629:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664871424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776167.319660:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664870912/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776167.319669:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.319673:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.319676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664870912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776167.319682:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871040:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776167.319725:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776167.319740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2f100 x1862032664871040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776167.319750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664871040/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776167.319757:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.319760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.319762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871040:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776167.319777:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776167.319795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776167.319804:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2e680 x1862032664871168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776167.319813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664871168/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776167.319820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.319823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.319825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776167.319830:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664871296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776167.320457:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776167.320475:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2df80 x1862032664871296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776167.320488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664871296/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776167.320496:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.320499:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.320502:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664871296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776167.320508:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664871424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776167.320530:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776167.320539:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776167/real 1775776167] req@ffff982974b2ca80 x1862032664871424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776167.320549:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664871424/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776167.320556:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776167.320559:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776167.320563:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664871424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776167.831171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776168.087201:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776168.855186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776169.111213:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776169.879166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776170.135200:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776170.903151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776171.159249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776171.927170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776172.183225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776172.439174:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439184:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776172.439188:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439198:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439210:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776172.439216:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776172.439252:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664871552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776172.439283:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439288:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776172.439290:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439294:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439298:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776172.439302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439312:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776172.439326:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664871680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776172.439331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664871552:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776172.439342:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439346:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776172.439348:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439352:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439356:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776172.439360:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776172.439378:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664871808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776172.439387:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439391:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776172.439392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439396:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439399:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776172.439404:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776172.439407:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439414:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776172.439427:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664871936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776172.439437:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776172.439441:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2d180 x1862032664871552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776172.439442:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776172.439446:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439455:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439460:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776172.439464:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664871552/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776172.439466:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776172.439478:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439483:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439487:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664871552:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776172.439492:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664872064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776172.439503:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439506:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776172.439508:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776172.439514:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00100000:3.0:1775776172.439514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871680:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776172.439518:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776172.439524:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439529:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776172.439541:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664872192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776172.439549:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776172.439559:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776172.439563:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776172.439565:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776172.439569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2e300 x1862032664871680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776172.439572:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17939 00000100:00080000:0.0:1775776172.439579:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776172.439585:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776172.439590:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776172.439591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664871680/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439603:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776172.439605:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664872320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776172.439609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439616:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776172.439625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776172.439643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776172.439651:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2d500 x1862032664871808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776172.439658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664871808/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439667:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776172.439673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871936:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776172.439690:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776172.439700:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2ca80 x1862032664871936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776172.439711:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664871936/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439718:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439721:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664871936:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776172.439730:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776172.439748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776172.439758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2df80 x1862032664872064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776172.439779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664872064/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776172.439807:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664872192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776172.439830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776172.439840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2e680 x1862032664872192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776172.439850:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664872192/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439858:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439861:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439863:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664872192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776172.439869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664872320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776172.439899:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776172.439909:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776172/real 1775776172] req@ffff982974b2f100 x1862032664872320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776172.439926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664872320/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776172.439934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776172.439936:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776172.439938:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664872320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776172.951180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776173.207252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776173.975278:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776174.231235:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776174.999161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776175.255334:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776176.023184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776176.279227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776177.047149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776177.303241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776177.559199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776177.559208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776177.559211:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776177.559221:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559233:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776177.559238:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776177.559275:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664872448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776177.559302:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776177.559306:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776177.559308:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776177.559312:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559315:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776177.559319:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776177.559328:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776177.559343:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664872576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776177.559353:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776177.559355:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664872448:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776177.559357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776177.559361:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776177.559366:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559369:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776177.559374:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776177.559378:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776177.559392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664872704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776177.559401:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776177.559410:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776177.559418:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776177.559420:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776177.559425:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559430:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776177.559434:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559438:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b6680 x1862032664872448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776177.559440:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776177.559456:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664872448/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776177.559460:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664872832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776177.559474:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776177.559477:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776177.559477:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776177.559479:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776177.559483:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776177.559487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00100000:3.0:1775776177.559488:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664872448:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776177.559493:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776177.559498:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776177.559503:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776177.559512:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775776177.559517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664872960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776177.559529:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776177.559532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776177.559536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776177.559538:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776177.559544:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559549:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776177.559551:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b4e00 x1862032664872576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776177.559557:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559564:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664872576/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776177.559565:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776177.559574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775776177.559582:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664873088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776177.559582:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776177.559591:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872704:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776177.559598:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776177.559601:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776177.559603:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776177.559607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776177.559610:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17945 00000100:00080000:2.0:1775776177.559615:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776177.559617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b7b80 x1862032664872704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776177.559623:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559629:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664872704/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776177.559631:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776177.559639:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775776177.559647:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664873216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776177.559653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776177.559665:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872832:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776177.559688:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559698:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b7100 x1862032664872832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776177.559709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664872832/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776177.559714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776177.559718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872832:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776177.559730:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872960:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776177.559743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b6a00 x1862032664872960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776177.559763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664872960/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776177.559768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776177.559772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664872960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776177.559775:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776177.559785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559790:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b4000 x1862032664873088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776177.559796:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664873088/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776177.559800:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776177.559804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776177.559807:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776177.559815:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776177.559819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776177/real 1775776177] req@ffff9829764b6680 x1862032664873216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776177.559825:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664873216/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776177.559829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776177.559831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776177.559832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776178.071169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776178.327207:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776179.095163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776179.351298:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776180.119174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776180.375240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776181.143146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776181.399237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776182.167199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776182.423294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776182.679254:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679274:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776182.679280:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679293:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679309:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679317:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776182.679368:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664873344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776182.679407:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776182.679416:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679420:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679431:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776182.679449:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664873344:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776182.679466:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664873472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776182.679478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679482:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776182.679484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679489:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679493:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679498:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679503:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776182.679516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664873600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776182.679520:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776182.679529:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679532:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776182.679534:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679538:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679546:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776182.679559:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe3480 x1862032664873344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776182.679570:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664873728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776182.679578:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679581:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776182.679583:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776182.679584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664873344/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776182.679588:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679592:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679597:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679601:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776182.679610:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664873856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776182.679616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776182.679617:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679619:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776182.679620:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679623:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679627:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776182.679630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776182.679631:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679636:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776182.679637:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664873344:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776182.679644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664873984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776182.679651:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776182.679653:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776182.679655:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776182.679658:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17950 00000100:00080000:1.0:1775776182.679660:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776182.679662:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776182.679664:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776182.679671:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664874112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776182.679677:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776182.679705:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe1880 x1862032664873472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664873472/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679732:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679734:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776182.679741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776182.679753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe2300 x1862032664873600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679765:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664873600/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776182.679780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873728:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776182.679797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe2680 x1862032664873728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664873728/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679844:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873728:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776182.679849:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776182.679863:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe2a00 x1862032664873856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664873856/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679882:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664873856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776182.679888:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776182.679897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe0380 x1862032664873984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679909:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664873984/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679913:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679915:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679916:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664873984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776182.679930:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664874112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776182.679938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776182.679944:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776182/real 1775776182] req@ffff982981fe0000 x1862032664874112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776182.679950:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664874112/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776182.679954:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776182.679956:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776182.679965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664874112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776183.191165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776183.447245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776184.215210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776184.471270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776185.239153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776185.495284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776186.263168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776186.519261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776187.287195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776187.543319:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776187.799210:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799219:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776187.799224:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799234:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799248:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799256:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776187.799306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664874240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776187.799336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776187.799341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799348:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799352:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776187.799376:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664874368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776187.799385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776187.799390:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664874240:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776187.799392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799401:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799404:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799407:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799413:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776187.799423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664874496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776187.799432:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799436:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776187.799437:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799441:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799453:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799457:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776187.799462:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776187.799476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664874624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776187.799487:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799490:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776187.799492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799495:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799499:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799502:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799506:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776187.799513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b5c00 x1862032664874240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776187.799517:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664874752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776187.799541:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776187.799544:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776187.799545:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799548:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799550:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799553:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799556:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776187.799556:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664874240/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776187.799566:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664874880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776187.799568:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799575:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776187.799578:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776187.799581:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664874240:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776187.799582:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776187.799585:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776187.799589:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17955 00000100:00080000:2.0:1775776187.799591:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776187.799593:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776187.799595:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776187.799601:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664875008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776187.799608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776187.799628:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799637:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b5180 x1862032664874368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799645:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664874368/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799650:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776187.799658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776187.799668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b6d80 x1862032664874496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799685:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664874496/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799692:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776187.799696:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874624:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776187.799707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799736:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b4a80 x1862032664874624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799750:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664874624/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799765:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874624:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776187.799771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776187.799800:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799811:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b5500 x1862032664874752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664874752/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799834:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799838:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664874752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776187.799841:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664874880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776187.799855:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b4700 x1862032664874880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664874880/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664874880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776187.799892:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776187.799901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776187.799906:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776187/real 1775776187] req@ffff9829764b5f80 x1862032664875008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776187.799912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664875008/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776187.799917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776187.799918:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776187.799921:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776188.311192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776188.567300:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776189.335166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776189.591248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776190.359170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776190.615278:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776191.383203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776191.639230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776192.407191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776192.663265:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776192.919279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776192.919297:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919309:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919324:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919332:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776192.919387:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664875136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776192.919412:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919415:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776192.919417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919421:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919430:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919441:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776192.919454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664875264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776192.919456:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664875136:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776192.919463:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919466:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776192.919468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919475:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919478:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919481:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776192.919490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664875392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776192.919497:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776192.919501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919504:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919507:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919511:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776192.919526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664875520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776192.919528:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776192.919533:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919536:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776192.919537:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919540:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919542:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919545:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776192.919557:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664875648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776192.919564:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776192.919566:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2fb80 x1862032664875136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776192.919567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776192.919569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919575:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919578:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776192.919585:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664875136/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776192.919588:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664875776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776192.919596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776192.919602:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776192.919605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776192.919605:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664875136:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776192.919607:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776192.919610:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17960 00000100:00080000:0.0:1775776192.919613:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776192.919615:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776192.919618:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776192.919642:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664875904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776192.919645:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776192.919669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2f480 x1862032664875264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.919691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664875264/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.919699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.919705:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776192.919710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776192.919729:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919740:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2c380 x1862032664875392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.919751:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664875392/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.919759:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.919763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776192.919768:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875520:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776192.919789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2f100 x1862032664875520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.919810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664875520/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.919818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.919824:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875520:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776192.919830:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776192.919849:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919859:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2e680 x1862032664875648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.919870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664875648/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.919877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.919883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664875648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776192.919888:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875776:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776192.919907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919917:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2df80 x1862032664875776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.919928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664875776/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.919935:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.919938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.919940:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776192.919969:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776192.919989:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776192.919999:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776192/real 1775776192] req@ffff982974b2ca80 x1862032664875904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776192.920017:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664875904/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776192.920025:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776192.920035:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776192.920039:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664875904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776193.431186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776193.687264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776194.455225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776194.711770:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776195.479203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776195.736127:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776196.503192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776196.759269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776197.527218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776197.783274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776198.039234:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039243:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776198.039252:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039265:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:3.0:1775776198.039279:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039285:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776198.039333:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664876032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776198.039364:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039367:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776198.039369:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039373:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:3.0:1775776198.039377:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039379:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039390:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776198.039404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664876160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776198.039416:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664876032:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776198.039416:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039424:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776198.039426:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039431:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:3.0:1775776198.039435:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039439:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776198.039453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664876288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776198.039460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039463:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776198.039464:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039467:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:3.0:1775776198.039470:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039472:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776198.039483:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664876416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776198.039493:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776198.039498:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039501:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776198.039502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039505:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:3.0:1775776198.039508:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039510:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039513:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776198.039520:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664876544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776198.039526:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776198.039529:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776198.039532:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00100000:1.0:1775776198.039554:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9dc00 x1862032664876032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776198.039557:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039564:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039568:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776198.039583:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664876672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776198.039591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664876032/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776198.039596:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776198.039605:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776198.039607:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776198.039608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776198.039612:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17965 00000100:00080000:1.0:1775776198.039614:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664876032:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776198.039619:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776198.039625:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776198.039630:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776198.039639:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664876800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776198.039647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776198.039667:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039678:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9ea00 x1862032664876160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664876160/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776198.039704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776198.039714:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039720:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9d880 x1862032664876288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039726:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664876288/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039731:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776198.039738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876416:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776198.039755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9df80 x1862032664876416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664876416/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876416:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776198.039795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776198.039814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9f480 x1862032664876544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039836:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664876544/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664876544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776198.039854:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664876672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776198.039868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9d180 x1862032664876672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664876672/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664876672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776198.039902:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664876800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776198.039911:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776198.039917:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776198/real 1775776198] req@ffff982980f9c000 x1862032664876800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776198.039923:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664876800/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776198.039928:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776198.039930:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776198.039932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664876800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776198.551192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776198.807295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776199.575199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776199.831267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776200.599201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776200.855269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776201.623160:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776201.879258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776202.647178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776202.903273:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776203.159223:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776203.159254:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159267:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159282:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159290:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776203.159335:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664876928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776203.159361:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159364:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776203.159365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159368:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159371:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159374:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776203.159392:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664877056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776203.159399:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664876928:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776203.159403:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159406:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776203.159407:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159411:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159413:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159416:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776203.159423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664877184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776203.159428:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776203.159432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159434:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159436:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159439:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776203.159465:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664877312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776203.159471:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159473:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776203.159474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159477:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159481:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776203.159493:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776203.159494:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664877440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776203.159502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159504:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776203.159505:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776203.159508:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159510:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159512:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776203.159522:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664877568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776203.159527:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776203.159529:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776203.159531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776203.159534:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe0000 x1862032664876928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776203.159536:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17970 00000100:00080000:1.0:1775776203.159541:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776203.159547:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776203.159552:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776203.159560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664877696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776203.159563:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664876928/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159581:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159589:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664876928:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776203.159611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776203.159627:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe0380 x1862032664877056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159646:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664877056/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159654:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776203.159667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776203.159687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe2a00 x1862032664877184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664877184/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159717:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776203.159725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877312:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776203.159745:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe2680 x1862032664877312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159765:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664877312/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877312:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776203.159792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877440:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776203.159805:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe2300 x1862032664877440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664877440/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776203.159828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664877568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776203.159837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159841:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe1880 x1862032664877568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664877568/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159855:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664877568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776203.159865:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664877696:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776203.159876:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776203.159882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776203/real 1775776203] req@ffff982981fe3480 x1862032664877696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776203.159892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664877696/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776203.159899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776203.159903:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776203.159906:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664877696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776203.671180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776203.927274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776204.695194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776204.951226:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776205.719206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776205.975276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776206.743189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776206.999249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776207.767196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776208.023384:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776208.279217:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279230:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776208.279236:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279249:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00080000:3.0:1775776208.279263:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279271:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776208.279310:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664877824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776208.279340:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279344:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776208.279345:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279351:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00080000:3.0:1775776208.279367:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279372:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776208.279388:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664877824:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776208.279397:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664877952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776208.279407:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279411:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776208.279412:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279416:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00080000:3.0:1775776208.279419:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279423:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279428:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776208.279438:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664878080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776208.279447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279450:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776208.279452:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279455:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00080000:3.0:1775776208.279466:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279470:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776208.279477:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776208.279492:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664878208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776208.279501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279504:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776208.279506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00080000:3.0:1775776208.279513:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776208.279533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664878336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776208.279534:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9ce00 x1862032664877824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776208.279558:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279562:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776208.279564:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279568:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00100000:1.0:1775776208.279572:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664877824/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776208.279574:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279581:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279588:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776208.279597:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664878464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776208.279601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776208.279605:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776208.279608:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776208.279609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776208.279610:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776208.279616:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17975 00000100:00100000:1.0:1775776208.279619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664877824:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776208.279620:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776208.279624:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776208.279630:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776208.279636:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664878592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776208.279655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776208.279676:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9e300 x1862032664877952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664877952/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664877952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776208.279706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776208.279716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9f100 x1862032664878080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664878080/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776208.279742:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878208:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776208.279759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9c000 x1862032664878208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279779:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664878208/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279787:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279790:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878208:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776208.279798:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776208.279817:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279828:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9d180 x1862032664878336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664878336/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279844:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776208.279852:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664878464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776208.279866:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9f480 x1862032664878464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664878464/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279883:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279887:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664878464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776208.279890:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664878592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776208.279898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776208.279903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776208/real 1775776208] req@ffff982980f9df80 x1862032664878592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776208.279909:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664878592/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776208.279914:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776208.279916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776208.279918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664878592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776208.791216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776209.047442:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776209.815196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776210.071257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776210.839204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776211.095276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776211.863205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776212.119277:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776212.887189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776213.143222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776213.399193:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399203:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776213.399208:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399218:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399232:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399239:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776213.399299:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664878720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776213.399331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776213.399338:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399343:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399347:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399352:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776213.399364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776213.399369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664878720:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776213.399385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664878848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776213.399393:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399396:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776213.399397:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399400:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399403:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399406:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776213.399409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776213.399417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664878976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776213.399423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399426:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776213.399427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399430:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399433:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399436:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776213.399440:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776213.399443:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776213.399452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664879104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776213.399458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399461:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776213.399462:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399465:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399467:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399471:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776213.399476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776213.399478:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2d500 x1862032664878720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776213.399488:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664879232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776213.399495:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664878720/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776213.399501:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776213.399510:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776213.399513:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776213.399515:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776213.399518:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:1.0:1775776213.399523:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776213.399524:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399532:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399532:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664878720:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776213.399539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776213.399549:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664879360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776213.399563:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776213.399564:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878848:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776213.399568:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776213.399571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776213.399574:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17980 00000100:00080000:0.0:1775776213.399577:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776213.399579:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776213.399583:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776213.399591:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776213.399594:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664879488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776213.399605:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2e300 x1862032664878848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664878848/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399625:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776213.399629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776213.399640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2d180 x1862032664878976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399651:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664878976/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399657:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399659:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399660:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664878976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776213.399663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879104:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776213.399673:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399678:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2ca80 x1862032664879104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664879104/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399694:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399698:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879104:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776213.399703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776213.399720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2df80 x1862032664879232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664879232/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399757:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399760:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399762:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776213.399774:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664879360:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776213.399789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2e680 x1862032664879360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664879360/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399817:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664879360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776213.399828:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664879488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776213.399846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776213.399857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776213/real 1775776213] req@ffff982974b2f100 x1862032664879488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776213.399867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664879488/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776213.399875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776213.399878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776213.399880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664879488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776213.911209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776214.167312:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776214.935206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776215.191308:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776215.959220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776216.215291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776216.983199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776217.239270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776218.007219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776218.263261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776218.519246:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519260:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776218.519266:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519280:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519295:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519301:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776218.519345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664879616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776218.519378:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776218.519384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519388:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519392:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519396:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776218.519423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664879744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776218.519431:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664879616:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776218.519434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519438:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776218.519441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519445:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519447:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519450:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519454:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776218.519463:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664879872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776218.519470:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519474:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776218.519476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519480:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519483:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519488:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519493:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776218.519505:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664880000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776218.519515:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775776218.519520:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776218.519522:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776218.519525:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519530:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519535:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519539:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519545:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776218.519559:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664880128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776218.519565:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519569:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775776218.519570:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b5f80 x1862032664879616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776218.519571:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776218.519577:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519581:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519587:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519593:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776218.519604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664879616/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776218.519606:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664880256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776218.519634:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776218.519634:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776218.519640:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776218.519643:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:2.0:1775776218.519644:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776218.519652:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664879616:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776218.519653:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17986 00000100:00080000:2.0:1775776218.519670:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776218.519676:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776218.519681:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776218.519684:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775776218.519698:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664880384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776218.519715:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.519731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b4700 x1862032664879744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.519744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664879744/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.519751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.519754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.519758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776218.519764:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776218.519783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.519794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b5500 x1862032664879872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.519804:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664879872/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.519811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.519814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.519817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664879872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776218.519821:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880000:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776218.519840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.519849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b4a80 x1862032664880000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.519858:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664880000/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.519866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.519869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.519871:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880000:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776218.519876:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880128:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776218.519894:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.519903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b6d80 x1862032664880128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.519913:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664880128/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.519920:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.519923:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.519926:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776218.519931:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664880256:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776218.519948:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.519957:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b5180 x1862032664880256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.519967:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664880256/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.519974:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.519977:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.519980:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664880256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776218.520000:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664880384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776218.520018:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776218.520027:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776218/real 1775776218] req@ffff9829764b5c00 x1862032664880384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776218.520037:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664880384/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776218.520045:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776218.520048:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776218.520051:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664880384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776219.031234:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776219.287279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776220.055149:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776220.311252:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776221.079192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776221.335219:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776222.103124:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776222.359259:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776223.127230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776223.383280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776223.639182:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639189:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776223.639193:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639200:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639211:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639216:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776223.639255:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664880512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639279:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639282:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776223.639284:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639287:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639289:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639292:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639305:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776223.639318:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664880512:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776223.639319:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664880640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639333:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776223.639337:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639340:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639343:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639345:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639348:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776223.639357:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664880768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639364:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639367:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776223.639368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639370:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639372:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639375:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776223.639386:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664880896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639395:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776223.639395:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776223.639401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776223.639402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639405:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639408:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639410:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639414:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776223.639420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664881024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639428:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776223.639429:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639432:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639435:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776223.639436:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe3b80 x1862032664880512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776223.639439:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639445:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776223.639452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664881152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776223.639457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776223.639460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776223.639461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776223.639468:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17991 00000100:00080000:1.0:1775776223.639472:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776223.639475:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776223.639479:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776223.639488:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664881280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776223.639488:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664880512/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639513:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639523:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664880512:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776223.639551:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776223.639586:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe1f80 x1862032664880640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664880640/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639636:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776223.639646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776223.639670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe0a80 x1862032664880768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664880768/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639702:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639708:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776223.639714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880896:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776223.639748:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe0e00 x1862032664880896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664880896/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639799:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664880896:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776223.639805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776223.639825:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639837:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe1c00 x1862032664881024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664881024/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639855:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776223.639859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664881152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776223.639872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639878:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe1500 x1862032664881152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639885:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664881152/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639890:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639892:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639894:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664881152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776223.639898:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664881280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776223.639907:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776223.639913:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776223/real 1775776223] req@ffff982981fe1180 x1862032664881280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776223.639919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664881280/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776223.639924:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776223.639926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776223.639928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664881280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776224.151209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776224.407236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776225.175227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776225.431282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776226.199200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776226.455222:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776227.223179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776227.479267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776228.247184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776228.503193:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776228.759211:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759222:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776228.759227:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759238:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759249:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759255:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776228.759295:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664881408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776228.759321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759324:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776228.759325:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759335:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776228.759344:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776228.759353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664881536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776228.759361:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664881408:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776228.759372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776228.759375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759379:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759382:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759384:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776228.759388:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776228.759398:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664881664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776228.759408:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776228.759414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759426:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776228.759432:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776228.759445:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664881792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776228.759459:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881536:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776228.759460:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759467:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776228.759469:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759477:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759480:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776228.759486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776228.759492:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776228.759499:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664881920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776228.759508:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759512:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776228.759514:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776228.759516:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881792:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776228.759522:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00080000:3.0:1775776228.759527:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759532:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776228.759538:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881920:192.168.201.118@tcp:8:kworker.0 00000008:00100000:3.0:1775776228.759539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776228.759557:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664882048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776228.759589:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776228.759593:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776228.759595:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776228.759602:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 17996 00000100:00100000:1.0:1775776228.759605:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882048:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776228.759611:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776228.759617:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776228.759624:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776228.759638:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664882176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776228.759668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776229.271277:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776229.527253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776230.295178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776230.551264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776231.319215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776231.575274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775776231.831441:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775776231.831490:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775776231.831508:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831549:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831561:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831572:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831582:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831592:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776231.831605:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775776231.831608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9ed80 x1862032664881792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831640:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664881792/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831653:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831659:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831663:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881792:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776231.831673:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9e680 x1862032664881920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664881920/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831701:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831704:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831707:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776231.831715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9ea00 x1862032664881536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831742:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664881536/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831758:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776231.831768:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9dc00 x1862032664881664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664881664/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831796:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831800:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664881664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776231.831810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9d880 x1862032664881408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664881408/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664881408:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776231.831870:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9fb80 x1862032664882176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664882176/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831897:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776231.831904:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776228/real 1775776231] req@ffff982980f9d500 x1862032664882048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776231.831915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664882048/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776231.831922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776231.831926:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776231.831929:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776232.343142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776232.599292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776233.367197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776233.623260:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776233.879207:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776233.879221:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776233.879226:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879239:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879255:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879263:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776233.879299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664882304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776233.879333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776233.879337:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776233.879339:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879347:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879351:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879362:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776233.879378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664882432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776233.879394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776233.879398:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664882304:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776233.879399:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776233.879402:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879414:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776233.879435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664882560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776233.879450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776233.879453:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776233.879455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879467:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776233.879485:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664882688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776233.879494:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776233.879496:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776233.879500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776233.879502:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879507:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879512:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776233.879534:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664882816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776233.879542:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9c380 x1862032664882304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776233.879544:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776233.879547:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776233.879549:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776233.879554:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879559:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776233.879566:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879570:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776233.879580:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664882304/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776233.879582:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664882944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776233.879610:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776233.879614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776233.879614:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776233.879615:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776233.879624:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776233.879624:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18001 00000100:00080000:3.0:1775776233.879630:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776233.879632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664882304:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776233.879637:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776233.879643:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776233.879654:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664883072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776233.879676:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776233.879709:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.879723:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9c700 x1862032664882432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.879741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664882432/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.879749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.879752:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.879755:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776233.879761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776233.879781:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.879792:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9f800 x1862032664882560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.879803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664882560/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.879811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.879814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.879817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776233.879823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882688:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776233.879844:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.879853:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9d500 x1862032664882688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.879863:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664882688/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.879874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.879877:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.879880:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882688:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776233.879885:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776233.879902:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.879927:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9fb80 x1862032664882816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.879938:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664882816/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.879946:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.879948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.879951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664882816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776233.879957:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776233.879976:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.879986:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9d880 x1862032664882944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.879997:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664882944/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.880006:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.880008:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.880012:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664882944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776233.880018:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776233.880036:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776233.880045:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776233/real 1775776233] req@ffff982980f9dc00 x1862032664883072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776233.880056:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664883072/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776233.880098:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776233.880103:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776233.880106:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776234.391208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776234.647309:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776235.415199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776235.671272:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776236.439206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776236.695276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776237.463187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776237.719289:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776238.487188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776238.743251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776238.999199:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999209:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776238.999214:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999225:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999239:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999246:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776238.999296:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664883200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999329:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776238.999335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999340:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999360:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776238.999373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664883328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776238.999386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999399:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999403:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776238.999413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664883456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999424:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776238.999426:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664883200:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776238.999428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776238.999430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999446:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999449:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999453:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776238.999461:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664883584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999466:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776238.999469:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999472:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999473:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999475:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776238.999485:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664883712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999490:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776238.999496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999498:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999500:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999503:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776238.999512:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664883840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776238.999516:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776238.999520:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776238.999522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776238.999525:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18006 00000100:00080000:3.0:1775776238.999528:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776238.999530:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776238.999533:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776238.999542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664883968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776238.999543:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999597:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9ea00 x1862032664883200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776238.999618:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664883200/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776238.999629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776238.999633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776238.999638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664883200:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776238.999646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776238.999671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999683:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9e680 x1862032664883328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776238.999696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664883328/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776238.999703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776238.999707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776238.999710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776238.999736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776238.999755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9ed80 x1862032664883456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776238.999777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664883456/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776238.999784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776238.999788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776238.999790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776238.999796:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883584:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776238.999814:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999860:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9df80 x1862032664883584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776238.999872:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664883584/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776238.999881:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776238.999885:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776238.999889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883584:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776238.999896:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776238.999919:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999931:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9f480 x1862032664883712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776238.999943:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664883712/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776238.999950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776238.999954:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776238.999957:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664883712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776238.999963:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776238.999982:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776238.999991:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776238] req@ffff982980f9d180 x1862032664883840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776239.000002:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664883840/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776239.000010:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776239.000013:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776239.000016:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776239.000022:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776239.000041:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776239.000052:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776238/real 1775776239] req@ffff982980f9c000 x1862032664883968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776239.000062:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664883968/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776239.000115:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776239.000118:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776239.000122:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664883968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776239.511188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776239.767274:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776240.535225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776240.791289:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776241.559166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776241.815284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776242.583188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776242.839275:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776243.607172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776243.863235:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776244.119181:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119191:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776244.119195:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776244.119204:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119223:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776244.119263:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664884096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776244.119293:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119296:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776244.119298:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776244.119302:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119306:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119310:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776244.119328:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776244.119331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664884096:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776244.119343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664884224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776244.119354:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119358:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776244.119360:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776244.119364:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119368:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119372:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776244.119378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776244.119386:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776244.119390:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664884352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776244.119400:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776244.119405:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776244.119409:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119413:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776244.119416:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9f100 x1862032664884096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776244.119417:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776244.119424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776244.119434:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664884096/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776244.119436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664884480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776244.119455:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776244.119456:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776244.119461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776244.119463:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776244.119466:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00100000:1.0:1775776244.119469:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664884096:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776244.119469:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119474:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776244.119478:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776244.119486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664884608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776244.119492:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884224:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776244.119504:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119508:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776244.119510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775776244.119514:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776244.119516:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119521:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119527:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776244.119528:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9e300 x1862032664884224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775776244.119535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776244.119540:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664884224/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119545:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119550:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776244.119550:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664884736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776244.119556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776244.119567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884352:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776244.119568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776244.119574:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776244.119575:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776244.119578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18011 00000100:00080000:3.0:1775776244.119582:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776244.119586:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776244.119589:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776244.119592:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776244.119598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9ce00 x1862032664884352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776244.119603:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664884864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776244.119620:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664884352/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119627:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119630:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776244.119632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776244.119636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884480:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776244.119649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776244.119654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9c000 x1862032664884480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776244.119661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664884480/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119666:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119668:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776244.119670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884480:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776244.119680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776244.119689:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776244.119693:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9d180 x1862032664884608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776244.119699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664884608/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119704:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119706:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776244.119707:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664884608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776244.119710:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664884736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776244.119718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776244.119726:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9f480 x1862032664884736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776244.119763:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664884736/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776244.119777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664884736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776244.119782:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664884864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776244.119802:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776244.119813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776244/real 1775776244] req@ffff982980f9df80 x1862032664884864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776244.119824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664884864/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776244.119831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776244.119834:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776244.119837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664884864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776244.631211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776244.887290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776245.655161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776245.911221:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776246.679188:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776246.935270:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776247.703271:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776247.959223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776248.727159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776248.983258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776249.239225:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239232:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776249.239235:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239243:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239252:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239257:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776249.239289:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664884992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239311:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239313:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776249.239314:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239317:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239319:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239321:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239327:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776249.239335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664885120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239342:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239344:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776249.239345:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239350:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239351:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776249.239361:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664885248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239370:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776249.239371:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239373:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239374:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239376:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239381:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776249.239387:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664884992:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776249.239393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664885376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239410:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239412:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776249.239413:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239415:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239417:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239419:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776249.239428:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664885504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239433:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239435:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776249.239436:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239438:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239440:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239442:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239445:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776249.239449:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664885632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776249.239454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776249.239456:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776249.239457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776249.239460:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18016 00000100:00080000:2.0:1775776249.239463:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776249.239466:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776249.239468:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776249.239474:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664885760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776249.239499:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239543:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b6680 x1862032664884992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239563:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664884992/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239574:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239579:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664884992:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776249.239591:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776249.239623:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239635:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b4000 x1862032664885120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239671:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664885120/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239680:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239683:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776249.239714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776249.239735:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239745:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b6a00 x1862032664885248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664885248/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239770:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239774:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776249.239783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885376:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776249.239801:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239809:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b7100 x1862032664885376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664885376/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885376:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776249.239840:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776249.239857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b7b80 x1862032664885504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664885504/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239884:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239887:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239890:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664885504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776249.239896:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664885632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776249.239916:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239924:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b4e00 x1862032664885632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239931:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664885632/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239936:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239940:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664885632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776249.239944:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664885760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776249.239956:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776249.239961:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776249/real 1775776249] req@ffff9829764b5c00 x1862032664885760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776249.239967:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664885760/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776249.239972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776249.239974:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776249.239976:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664885760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776249.751201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776250.007261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776250.775226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776251.031287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776251.799198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776252.055323:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776252.823167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776253.079266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776253.847167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776254.103257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776254.359165:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776254.359174:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776254.359178:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359187:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359199:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359205:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776254.359236:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664885888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359265:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776254.359268:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776254.359270:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359274:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359278:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359282:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776254.359291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776254.359310:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664886016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359321:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776254.359326:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664885888:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776254.359326:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776254.359328:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359336:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359340:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359344:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776254.359350:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776254.359362:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664886144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359371:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776254.359374:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776254.359376:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359379:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359383:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359387:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776254.359392:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775776254.359393:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776254.359406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664886272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776254.359418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776254.359420:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359424:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359433:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776254.359438:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776254.359443:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b5180 x1862032664885888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776254.359452:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664886400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359463:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776254.359467:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664885888/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776254.359467:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776254.359470:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359476:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359481:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359487:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776254.359488:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775776254.359494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776254.359495:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359502:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664885888:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776254.359511:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664886528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776254.359520:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776254.359523:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775776254.359526:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886016:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776254.359526:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776254.359534:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18021 00000100:00080000:2.0:1775776254.359539:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776254.359543:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776254.359549:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776254.359567:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359582:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b6d80 x1862032664886016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776254.359582:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664886656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776254.359617:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664886016/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359632:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776254.359638:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776254.359658:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b4a80 x1862032664886144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776254.359677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664886144/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359690:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359692:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359695:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776254.359700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886272:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776254.359719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b5500 x1862032664886272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776254.359737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664886272/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359744:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359750:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886272:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776254.359755:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776254.359772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b4700 x1862032664886400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776254.359792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664886400/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359804:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776254.359809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664886528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776254.359825:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359834:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b5f80 x1862032664886528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776254.359845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664886528/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359857:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664886528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776254.359874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664886656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776254.359892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776254.359902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776254/real 1775776254] req@ffff9829764b5180 x1862032664886656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776254.359911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664886656/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776254.359918:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776254.359921:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776254.359924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664886656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776254.871205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776255.127245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776255.895174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776256.151239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776256.919166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776257.175299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776257.943173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776258.199266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776258.967161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776259.223241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776259.479193:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479200:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776259.479204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479217:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479229:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479235:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776259.479278:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664886784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479301:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479304:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776259.479305:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479308:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479311:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479314:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479321:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776259.479339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664886912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479350:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479354:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776259.479356:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776259.479359:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664886784:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776259.479363:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479370:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479373:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776259.479386:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664887040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479393:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776259.479397:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479399:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479402:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479404:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776259.479415:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664887168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479421:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479423:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776259.479424:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479426:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479428:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479432:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776259.479434:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479435:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776259.479443:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664887296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479455:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776259.479461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479474:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00080000:0.0:1775776259.479478:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479481:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479485:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776259.479487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2c380 x1862032664886784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776259.479494:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664887424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776259.479506:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776259.479508:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776259.479510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776259.479514:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18026 00000100:00100000:1.0:1775776259.479514:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664886784/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776259.479519:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776259.479525:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776259.479531:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776259.479532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776259.479538:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664887552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776259.479542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664886784:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776259.479568:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776259.479592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479602:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2f480 x1862032664886912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479610:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664886912/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479616:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479618:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664886912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776259.479623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776259.479635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2fb80 x1862032664887040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479646:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664887040/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776259.479662:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887168:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776259.479671:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479675:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2f100 x1862032664887168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664887168/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479686:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479687:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887168:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776259.479691:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776259.479700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479704:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2e680 x1862032664887296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664887296/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479715:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776259.479720:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664887424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776259.479728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479733:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2df80 x1862032664887424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664887424/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479744:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664887424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776259.479748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664887552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776259.479757:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776259.479761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776259/real 1775776259] req@ffff982974b2ca80 x1862032664887552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776259.479767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664887552/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776259.479771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776259.479773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776259.479774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664887552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776259.991173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776260.247263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776261.015125:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776261.271214:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776262.039251:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776262.295275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776263.063189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776263.319240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776264.087122:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776264.343441:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776264.599237:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599248:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776264.599253:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776264.599265:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599279:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599287:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776264.599333:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664887680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599363:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599367:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776264.599368:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776264.599397:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599401:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599405:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776264.599416:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776264.599429:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664887808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599441:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776264.599443:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776264.599447:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599449:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599452:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776264.599456:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776264.599464:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664887936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599478:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599482:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776264.599484:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776264.599487:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599490:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599495:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599497:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664887680:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775776264.599500:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776264.599514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664888064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776264.599528:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776264.599532:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599541:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599545:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776264.599550:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776264.599560:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664888192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776264.599571:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776264.599572:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775776264.599577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776264.599578:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599584:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776264.599588:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776264.599594:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776264.599605:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664888320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776264.599613:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776264.599617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9ed80 x1862032664887680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776264.599629:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776264.599631:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776264.599634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664887680/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776264.599637:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18032 00000100:00080000:3.0:1775776264.599644:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776264.599648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776264.599649:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776264.599654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775776264.599656:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776264.599658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664887680:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776264.599671:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664888448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776264.599679:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776264.599700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599710:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9e680 x1862032664887808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664887808/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599724:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599727:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599729:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776264.599732:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776264.599742:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9ea00 x1862032664887936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599755:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664887936/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599760:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599762:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664887936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776264.599787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888064:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776264.599798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9dc00 x1862032664888064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599810:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664888064/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599817:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599819:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888064:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776264.599822:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776264.599830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9d880 x1862032664888192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664888192/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599850:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599852:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599854:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776264.599857:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664888320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776264.599864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599869:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9fb80 x1862032664888320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664888320/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664888320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776264.599885:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664888448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776264.599893:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776264.599898:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776264/real 1775776264] req@ffff982980f9d500 x1862032664888448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776264.599904:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664888448/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776264.599909:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776264.599910:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776264.599913:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664888448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776265.111233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776265.367205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776266.135209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776266.391284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776267.159211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776267.415235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776268.183201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776268.439268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776269.207630:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776269.463310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776269.719224:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719236:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776269.719241:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719266:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719282:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719304:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776269.719353:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664888576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776269.719379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776269.719384:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719388:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719392:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719395:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776269.719415:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664888704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776269.719422:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719425:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776269.719426:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664888576:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776269.719428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719437:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719439:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719442:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776269.719449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664888832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776269.719455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719458:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776269.719459:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719461:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719463:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719465:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719468:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776269.719484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664888960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776269.719492:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719496:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776269.719498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719502:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719506:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776269.719513:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776269.719532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664889088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776269.719540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719542:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776269.719544:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719547:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719550:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776269.719553:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719557:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776269.719565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664889216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776269.719565:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe1180 x1862032664888576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776269.719587:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776269.719591:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776269.719593:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776269.719598:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18037 00000100:00080000:1.0:1775776269.719603:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776269.719604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664888576/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776269.719609:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776269.719618:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776269.719622:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776269.719632:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664889344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776269.719638:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664888576:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776269.719668:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776269.719693:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.719706:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe1500 x1862032664888704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.719719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664888704/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.719727:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719730:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.719733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776269.719738:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776269.719769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.719781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe1c00 x1862032664888832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.719792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664888832/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.719799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719802:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.719805:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776269.719811:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888960:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776269.719831:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.719842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe0e00 x1862032664888960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.719853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664888960/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.719861:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719864:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.719866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664888960:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776269.719871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889088:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776269.719890:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.719900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe0a80 x1862032664889088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.719911:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664889088/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.719919:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719922:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.719925:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776269.719931:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664889216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776269.719951:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.719962:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe1f80 x1862032664889216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.719973:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664889216/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.719981:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.719984:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.719987:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664889216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776269.719992:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664889344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776269.720011:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776269.720021:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776269/real 1775776269] req@ffff982981fe3b80 x1862032664889344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776269.720038:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664889344/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776269.720046:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776269.720049:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776269.720053:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664889344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776270.231187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776270.487294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776271.255147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776271.511292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776272.279152:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776272.535294:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776273.303218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776273.559297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776274.327198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776274.583294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776274.839231:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776274.839241:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776274.839245:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776274.839256:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00080000:1.0:1775776274.839269:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839278:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776274.839320:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664889472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776274.839351:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776274.839355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776274.839357:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776274.839362:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00080000:1.0:1775776274.839366:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839370:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839383:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776274.839398:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664889600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776274.839411:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664889472:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776274.839431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776274.839435:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776274.839436:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776274.839442:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00080000:1.0:1775776274.839447:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839452:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776274.839471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664889728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776274.839490:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776274.839491:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776274.839497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776274.839499:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776274.839505:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00080000:1.0:1775776274.839510:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839515:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776274.839529:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe3480 x1862032664889472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776274.839535:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664889856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776274.839549:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776274.839549:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664889472/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776274.839559:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776274.839564:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776274.839567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776274.839571:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00080000:3.0:1775776274.839574:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776274.839579:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776274.839579:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664889472:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776274.839588:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839594:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776274.839604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776274.839610:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664889984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776274.839625:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776274.839629:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776274.839633:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776274.839636:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776274.839642:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00100000:3.0:1775776274.839646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe1880 x1862032664889600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776274.839649:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839659:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839668:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776274.839675:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664889600/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776274.839683:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664890112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776274.839696:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776274.839698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776274.839704:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776274.839705:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776274.839709:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776274.839711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889600:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776274.839718:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18042 00000100:00100000:3.0:1775776274.839721:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889728:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776274.839727:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776274.839733:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776274.839740:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776274.839746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776274.839757:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664890240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776274.839758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe2300 x1862032664889728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776274.839797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664889728/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776274.839809:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776274.839813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776274.839818:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776274.839824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889856:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776274.839848:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776274.839860:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe2680 x1862032664889856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776274.839869:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664889856/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776274.839874:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776274.839876:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776274.839879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889856:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776274.839892:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776274.839906:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776274.839912:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe2a00 x1862032664889984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776274.839920:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664889984/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776274.839925:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776274.839927:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776274.839929:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664889984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776274.839932:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664890112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776274.839942:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776274.839952:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe0380 x1862032664890112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776274.839963:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664890112/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776274.839972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776274.839975:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776274.839978:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664890112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776274.839984:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664890240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776274.840005:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776274.840017:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776274/real 1775776274] req@ffff982981fe0000 x1862032664890240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776274.840029:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664890240/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776274.840038:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776274.840041:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776274.840045:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664890240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776275.351209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776275.607285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776276.375220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776276.631292:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776277.399234:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776277.655337:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776278.423176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776278.679251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776279.447180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776279.703270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776279.959242:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959256:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776279.959262:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776279.959275:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959290:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959297:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776279.959345:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664890368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776279.959379:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776279.959385:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776279.959389:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959393:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959397:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776279.959421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664890496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776279.959431:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664890368:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776279.959434:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776279.959441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776279.959446:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959449:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959453:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959458:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776279.959472:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664890624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776279.959487:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959490:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776279.959493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776279.959497:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959500:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959504:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776279.959509:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959513:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776279.959529:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664890752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776279.959538:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959542:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776279.959544:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776279.959548:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2d180 x1862032664890368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776279.959550:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959557:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959562:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959567:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776279.959568:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664890368/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959579:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776279.959582:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664890880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776279.959584:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664890368:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776279.959603:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776279.959606:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776279.959613:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776279.959616:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890496:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776279.959621:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959626:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959631:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776279.959635:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959636:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776279.959645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2e300 x1862032664890496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776279.959651:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664891008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776279.959658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664890496/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776279.959673:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776279.959676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776279.959679:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776279.959682:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890496:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776279.959683:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776279.959688:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890624:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776279.959690:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18047 00000100:00080000:0.0:1775776279.959696:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776279.959701:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776279.959705:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776279.959706:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776279.959715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2d500 x1862032664890624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776279.959721:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664891136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776279.959733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664890624/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776279.959755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776279.959773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890752:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776279.959793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776279.959800:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2ca80 x1862032664890752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776279.959807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664890752/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959812:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776279.959814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890752:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776279.959820:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776279.959830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776279.959835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2df80 x1862032664890880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776279.959841:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664890880/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776279.959846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664890880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776279.959863:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776279.959880:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776279.959890:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2e680 x1862032664891008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776279.959901:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664891008/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959908:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776279.959911:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959914:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776279.959919:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776279.959932:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776279.959938:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776279/real 1775776279] req@ffff982974b2f100 x1862032664891136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776279.959945:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664891136/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776279.959950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776279.959951:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776279.959953:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776280.471162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776280.727257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776281.495181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776281.751297:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776282.519246:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776282.775212:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776283.543260:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776283.799288:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776284.567197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776284.823348:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776285.079155:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079163:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776285.079167:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079175:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079185:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776285.079190:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776285.079226:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664891264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776285.079250:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079253:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776285.079255:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079258:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079260:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776285.079263:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079271:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776285.079279:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664891392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776285.079286:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079289:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776285.079290:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079293:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079295:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776285.079297:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079301:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776285.079304:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664891264:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776285.079314:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664891520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776285.079323:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079326:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776285.079328:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079331:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079334:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776285.079336:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079341:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776285.079348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664891648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776285.079354:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079357:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776285.079358:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079364:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18052 00000800:00000100:3.0:1775776285.079366:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776285.079368:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776285.079371:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079374:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776285.079381:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664891776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776285.079388:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079391:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776285.079392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776285.079396:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079400:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776285.079400:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2fb80 x1862032664891264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776285.079407:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776285.079423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664891904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776285.079425:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664891264/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776285.079450:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776285.079453:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776285.079453:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776285.079457:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776285.079461:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776285.079462:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18052 00000100:00080000:0.0:1775776285.079466:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776285.079468:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664891264:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776285.079471:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776285.079478:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776285.079483:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776285.079487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664892032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776285.079504:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079514:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2f480 x1862032664891392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079523:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664891392/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079529:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079531:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079533:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776285.079552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776285.079564:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079571:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2c380 x1862032664891520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079578:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664891520/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079586:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079588:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776285.079592:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891648:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776285.079602:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2f100 x1862032664891648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664891648/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079622:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079624:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891648:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776285.079627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776285.079637:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2e680 x1862032664891776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664891776/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079662:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664891776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776285.079665:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891904:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776285.079674:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2df80 x1862032664891904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664891904/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079691:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079693:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079697:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664891904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776285.079702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776285.079720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776285.079731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776285/real 1775776285] req@ffff982974b2ca80 x1862032664892032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776285.079743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664892032/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776285.079755:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776285.079758:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776285.079761:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776285.591164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776285.847256:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776286.615185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776286.871249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776287.639166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776287.895224:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776288.663158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776288.919236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776289.687176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776289.943242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776290.199156:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776290.199166:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776290.199177:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199188:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199200:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199207:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776290.199255:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664892160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776290.199285:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776290.199295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776290.199298:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199303:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199307:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199312:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199322:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664892160:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776290.199323:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776290.199342:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664892288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776290.199353:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776290.199356:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776290.199357:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199361:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199365:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199369:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776290.199374:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776290.199388:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664892416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776290.199406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776290.199410:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776290.199412:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776290.199415:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199425:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199429:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776290.199436:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776290.199451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664892544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776290.199455:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2d500 x1862032664892160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776290.199463:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776290.199470:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776290.199472:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199477:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199480:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199483:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776290.199487:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776290.199490:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664892160/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776290.199496:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664892672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776290.199510:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776290.199510:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776290.199516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776290.199517:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776290.199520:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776290.199523:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199524:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199526:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776290.199528:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776290.199536:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664892160:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776290.199537:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664892800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776290.199550:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776290.199553:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776290.199554:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776290.199558:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18057 00000100:00080000:0.0:1775776290.199559:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776290.199562:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776290.199564:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776290.199569:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664892928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776290.199574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776290.199603:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199620:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2e300 x1862032664892288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664892288/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.199643:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.199647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.199660:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776290.199666:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892416:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776290.199687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2d180 x1862032664892416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199709:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664892416/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.199717:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.199720:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.199723:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776290.199729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892544:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776290.199750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2ca80 x1862032664892544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664892544/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.199777:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.199780:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.199784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892544:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776290.199789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776290.199807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2df80 x1862032664892672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199827:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664892672/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.199833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.199837:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.199840:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664892672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776290.199845:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776290.199864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2e680 x1862032664892800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664892800/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.199894:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.199897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.199900:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776290.199919:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776290.199960:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776290.199984:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776290/real 1775776290] req@ffff982974b2f100 x1862032664892928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776290.199995:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664892928/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776290.200003:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776290.200006:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776290.200009:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664892928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776290.711206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776290.967288:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776291.735205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776291.991269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776292.759184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776293.015249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776293.783172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776294.039253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776294.807186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776295.063266:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776295.319133:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319144:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776295.319149:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319159:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319171:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776295.319178:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776295.319216:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664893056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776295.319243:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319256:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776295.319259:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319264:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319269:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776295.319274:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664893056:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776295.319277:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776295.319291:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776295.319308:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664893184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776295.319320:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319324:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776295.319326:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319330:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776295.319338:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776295.319343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776295.319356:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664893312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776295.319368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776295.319369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893184:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776295.319375:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776295.319378:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319389:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319393:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776295.319397:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776295.319403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776295.319426:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664893440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776295.319429:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893312:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776295.319447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319451:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776295.319454:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319458:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319464:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776295.319469:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776295.319473:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893440:192.168.201.118@tcp:8:kworker.0 00000008:00100000:0.0:1775776295.319477:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776295.319494:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664893568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776295.319517:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319521:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776295.319523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776295.319527:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319532:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776295.319538:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893568:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775776295.319539:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776295.319548:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776295.319562:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664893696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776295.319574:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776295.319578:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776295.319580:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776295.319583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664893696:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776295.319588:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18062 00000100:00080000:0.0:1775776295.319594:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776295.319598:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776295.319603:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776295.319618:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664893824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776295.319649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664893824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776295.831163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776296.087230:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776296.855195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776297.111283:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776297.879199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776298.135226:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775776298.391397:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775776298.391438:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775776298.391462:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391519:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391530:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391538:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391547:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391557:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776298.391570:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776298.391598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2ca80 x1862032664893824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391627:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664893824/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391640:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664893824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776298.391656:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2df80 x1862032664893696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391668:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664893696/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391679:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664893696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776298.391687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2e680 x1862032664893568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664893568/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391705:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776298.391717:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2f100 x1862032664893440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664893440/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893440:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776298.391751:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2fb80 x1862032664893312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664893312/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391772:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391775:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776298.391782:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2f480 x1862032664893184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664893184/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664893184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776298.391847:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776295/real 1775776298] req@ffff982974b2c380 x1862032664893056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776298.391861:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664893056/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776298.391871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776298.391875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776298.391879:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664893056:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776298.903175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776299.159263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776299.927185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776300.183223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776300.439172:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439182:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776300.439186:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776300.439196:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439208:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776300.439214:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776300.439256:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664893952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776300.439283:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439287:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776300.439289:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776300.439293:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439296:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776300.439300:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439310:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776300.439322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664894080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776300.439332:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439336:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776300.439339:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776300.439341:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664893952:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776300.439345:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439365:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776300.439370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439376:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776300.439387:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664894208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776300.439396:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439400:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776300.439402:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776300.439406:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439410:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776300.439423:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776300.439427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776300.439445:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664894336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776300.439454:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439459:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776300.439461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776300.439465:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439469:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776300.439472:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9f800 x1862032664893952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776300.439476:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776300.439498:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664894464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776300.439499:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664893952/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776300.439523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776300.439526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776300.439527:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776300.439532:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776300.439535:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776300.439539:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439543:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776300.439545:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664893952:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776300.439550:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439555:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776300.439569:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664894592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776300.439580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894080:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776300.439580:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776300.439587:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776300.439589:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776300.439594:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18067 00000100:00080000:3.0:1775776300.439598:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776300.439601:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776300.439605:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776300.439614:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776300.439618:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664894720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776300.439630:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9c700 x1862032664894080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.439642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664894080/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.439649:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.439653:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.439655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776300.439660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776300.441187:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776300.441212:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9c380 x1862032664894208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.441236:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664894208/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.441245:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.441249:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.441259:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776300.441268:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894336:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776300.441295:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776300.441319:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9d500 x1862032664894336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.441335:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664894336/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.441343:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.441346:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.441349:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894336:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776300.441356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776300.441382:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776300.441393:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9fb80 x1862032664894464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.441417:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664894464/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.441427:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.441429:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.441431:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776300.441452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664894592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776300.441466:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776300.441474:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9d880 x1862032664894592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.441480:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664894592/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.441485:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.441487:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.441488:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664894592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776300.441492:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664894720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776300.441500:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776300.441506:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776300/real 1775776300] req@ffff982980f9dc00 x1862032664894720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776300.441511:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664894720/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776300.441516:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776300.441518:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776300.441520:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664894720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776300.951183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776301.207284:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776301.975170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776302.231220:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776302.999211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776303.255241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776304.023176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776304.279226:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776305.047233:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776305.303258:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776305.559190:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559197:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776305.559200:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559208:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559218:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559223:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776305.559259:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664894848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776305.559286:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559290:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776305.559292:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559296:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559299:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776305.559312:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776305.559325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664894976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776305.559335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776305.559340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559346:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00100000:1.0:1775776305.559348:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664894848:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776305.559350:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559358:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776305.559363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776305.559375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664895104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776305.559386:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559390:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776305.559392:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559396:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559400:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559404:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776305.559410:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776305.559421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776305.559422:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664895232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776305.559430:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559434:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776305.559435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559438:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559447:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559451:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776305.559454:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776305.559462:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b5c00 x1862032664894848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776305.559462:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664895360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776305.559481:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559485:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776305.559487:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559489:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559492:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776305.559495:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559497:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664894848/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775776305.559501:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776305.559509:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559514:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775776305.559518:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664895488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776305.559519:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664894848:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776305.559530:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894976:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776305.559536:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776305.559540:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776305.559542:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776305.559547:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18073 00000100:00080000:2.0:1775776305.559552:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775776305.559554:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776305.559560:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776305.559565:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776305.559569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b4e00 x1862032664894976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776305.559577:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664895616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776305.559587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664894976/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559596:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559600:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559604:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664894976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776305.559629:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776305.559651:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559678:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b7b80 x1862032664895104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776305.559691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664895104/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559703:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559706:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776305.559711:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895232:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776305.559732:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559743:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b7100 x1862032664895232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776305.559753:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664895232/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559758:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559761:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559763:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895232:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776305.559766:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776305.559782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559791:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b6a00 x1862032664895360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776305.559808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664895360/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776305.559826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664895488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776305.559845:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559854:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b4000 x1862032664895488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776305.559865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664895488/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559873:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559875:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559878:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664895488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776305.559883:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664895616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776305.559901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776305.559911:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776305/real 1775776305] req@ffff9829764b6680 x1862032664895616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776305.559921:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664895616/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776305.559934:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776305.559938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776305.559941:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664895616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776306.071193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776306.327301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776307.095260:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776307.351280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776308.119224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776308.375351:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776309.143178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776309.399192:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776310.167186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776310.423330:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776310.679230:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679244:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776310.679250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679263:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679277:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679284:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776310.679330:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664895744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776310.679366:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679370:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776310.679372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679376:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679379:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679383:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776310.679409:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664895872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776310.679417:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776310.679421:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679424:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679427:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679429:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679431:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776310.679439:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664896000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776310.679452:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776310.679456:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679463:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00100000:3.0:1775776310.679467:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664895744:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776310.679468:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679476:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776310.679495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664896128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776310.679501:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679504:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776310.679507:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679509:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679512:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679515:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679518:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776310.679526:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664896256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776310.679531:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679533:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776310.679534:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679536:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679538:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679542:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679547:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776310.679564:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664896384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776310.679566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776310.679589:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776310.679593:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776310.679595:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776310.679600:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18078 00000100:00080000:1.0:1775776310.679606:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776310.679611:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776310.679617:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776310.679628:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe0000 x1862032664895744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776310.679629:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664896512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776310.679700:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664895744/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.679714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.679719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.679724:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664895744:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776310.679733:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776310.679759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.679772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe0380 x1862032664895872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.679784:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664895872/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.679793:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.679796:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.679800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664895872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776310.679826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776310.679847:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.679858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe2a00 x1862032664896000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.679870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664896000/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.679878:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.679881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.679885:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776310.679891:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896128:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776310.679912:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.679924:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe2680 x1862032664896128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.679936:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664896128/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.679944:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.679948:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.679951:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896128:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776310.679958:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776310.679976:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.679987:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe2300 x1862032664896256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.679999:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664896256/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.680006:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.680009:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.680012:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776310.680018:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664896384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776310.680034:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.680043:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe1880 x1862032664896384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.680053:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664896384/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.680060:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.680102:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.680106:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664896384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776310.680111:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664896512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776310.680143:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776310.680153:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776310/real 1775776310] req@ffff982981fe3480 x1862032664896512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776310.680163:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664896512/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776310.680181:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776310.680183:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776310.680185:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664896512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776311.191190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776311.447465:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776312.215202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776312.471286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776313.239283:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776313.495242:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776314.263192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776314.519252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776315.287159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776315.543251:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776315.799203:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776315.799213:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776315.799217:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799228:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:0.0:1775776315.799240:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799247:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776315.799283:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664896640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776315.799316:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776315.799319:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776315.799321:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799326:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:0.0:1775776315.799329:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799334:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776315.799367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664896768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776315.799368:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664896640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776315.799385:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776315.799388:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776315.799390:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:0.0:1775776315.799399:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799403:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776315.799421:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664896896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776315.799431:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776315.799436:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776315.799437:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776315.799441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799446:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:0.0:1775776315.799451:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799463:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776315.799499:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2d180 x1862032664896640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776315.799509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664897024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776315.799520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664896640/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776315.799521:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776315.799530:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776315.799534:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776315.799539:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776315.799541:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:3.0:1775776315.799547:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776315.799548:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776315.799552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664896640:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776315.799555:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799564:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776315.799577:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664897152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776315.799580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896768:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776315.799597:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776315.799600:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776315.799602:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799607:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18083 00000800:00000100:3.0:1775776315.799611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776315.799612:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799618:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799624:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776315.799628:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2e300 x1862032664896768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776315.799639:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664897280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776315.799644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664896768/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896768:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776315.799678:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776315.799681:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896896:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776315.799682:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776315.799685:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776315.799689:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18083 00000100:00080000:0.0:1775776315.799693:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776315.799696:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776315.799700:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776315.799703:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776315.799716:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664897408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776315.799716:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2d500 x1862032664896896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776315.799746:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664896896/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799754:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664896896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776315.799761:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897024:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776315.799773:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776315.799778:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2c380 x1862032664897024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776315.799785:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664897024/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897024:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776315.799802:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776315.799812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776315.799818:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2f480 x1862032664897152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776315.799824:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664897152/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799828:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776315.799834:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664897280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776315.799842:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776315.799846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2fb80 x1862032664897280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776315.799853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664897280/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664897280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776315.799862:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664897408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776315.799870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776315.799874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776315/real 1775776315] req@ffff982974b2f100 x1862032664897408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776315.799880:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664897408/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776315.799885:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776315.799887:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776315.799889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664897408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776316.311178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776316.567219:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776317.335135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776317.591245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776318.359158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776318.615256:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776319.383167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776319.639186:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776320.407171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776320.663238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776320.919246:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919253:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776320.919257:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919264:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919275:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919279:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776320.919312:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664897536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776320.919340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919347:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919354:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776320.919362:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664897664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919369:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919371:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776320.919372:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919375:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919376:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919378:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919380:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776320.919386:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664897792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776320.919423:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776320.919423:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664897536:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776320.919429:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919437:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919440:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776320.919454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664897920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919464:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776320.919466:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919468:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919470:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919473:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776320.919484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664898048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919492:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776320.919493:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919495:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919507:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919518:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776320.919525:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776320.919531:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664898176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776320.919538:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776320.919540:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776320.919541:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776320.919545:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18088 00000100:00080000:2.0:1775776320.919550:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776320.919552:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776320.919557:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776320.919564:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664898304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776320.919585:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b6680 x1862032664897536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919607:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664897536/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919619:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919623:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919627:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664897536:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776320.919636:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776320.919661:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919673:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b4000 x1862032664897664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664897664/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776320.919731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776320.919750:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b6a00 x1862032664897792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919772:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664897792/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919780:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919786:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776320.919793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897920:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776320.919813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919823:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b7100 x1862032664897920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919834:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664897920/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664897920:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776320.919853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776320.919872:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919882:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b7b80 x1862032664898048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919892:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664898048/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919904:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776320.919910:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664898176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776320.919929:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919939:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b4e00 x1862032664898176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.919950:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664898176/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.919958:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.919961:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.919964:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664898176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776320.919970:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664898304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776320.919986:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776320.919994:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776320/real 1775776320] req@ffff9829764b5c00 x1862032664898304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776320.920006:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664898304/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776320.920011:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776320.920014:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776320.920017:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664898304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776321.431196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776321.687227:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776322.455179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776322.711200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776323.479167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776323.735176:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776324.503165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776324.759221:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776325.527163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776325.783217:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776326.039192:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776326.039203:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039224:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039230:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776326.039267:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664898432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039291:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039295:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776326.039297:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039301:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039316:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039320:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776326.039330:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776326.039342:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664898432:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776326.039345:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664898560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039355:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776326.039358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039361:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039366:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776326.039369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776326.039378:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664898688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039383:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039386:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776326.039387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039389:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039391:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039393:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776326.039397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776326.039401:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776326.039405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664898816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039414:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776326.039415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039419:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039422:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776326.039424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776326.039432:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2e680 x1862032664898432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776326.039433:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664898944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039455:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039457:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776326.039458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776326.039461:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664898432/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776326.039464:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039470:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039474:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776326.039475:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776326.039477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776326.039481:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039484:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664898432:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776326.039486:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664899072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776326.039494:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776326.039496:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776326.039498:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776326.039500:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18093 00000100:00080000:0.0:1775776326.039501:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776326.039503:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776326.039505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776326.039508:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776326.039513:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664899200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776326.039526:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039536:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2df80 x1862032664898560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039544:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664898560/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039550:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039552:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039554:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776326.039557:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776326.039566:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039571:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2ca80 x1862032664898688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664898688/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039582:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039584:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039585:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776326.039588:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898816:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776326.039598:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039603:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2f100 x1862032664898816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039609:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664898816/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039615:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039617:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898816:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776326.039620:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776326.039628:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039633:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2fb80 x1862032664898944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039638:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664898944/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039644:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039646:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664898944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776326.039648:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776326.039656:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039661:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2f480 x1862032664899072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664899072/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039671:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039672:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776326.039683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776326.039691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776326.039697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776326/real 1775776326] req@ffff982974b2c380 x1862032664899200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776326.039710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664899200/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776326.039715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776326.039717:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776326.039718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776326.551202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776326.807246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776327.575133:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776327.831216:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776328.599191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776328.855203:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776329.623192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776329.879274:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776330.647163:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776330.903259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776331.159182:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159191:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776331.159199:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159208:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159228:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159234:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776331.159269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664899328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159291:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159293:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776331.159295:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159298:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159300:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159303:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776331.159311:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776331.159320:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664899456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159332:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776331.159332:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664899328:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776331.159334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159343:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159346:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776331.159349:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776331.159359:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664899584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776331.159370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159372:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159385:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159388:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776331.159392:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776331.159396:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776331.159404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664899712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159411:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159414:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776331.159415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159421:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159423:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776331.159426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776331.159428:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b5180 x1862032664899328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776331.159435:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664899840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776331.159446:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664899328/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776331.159447:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776331.159451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159459:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00080000:2.0:1775776331.159463:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776331.159463:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775776331.159468:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776331.159470:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:2.0:1775776331.159473:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776331.159476:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664899328:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776331.159484:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664899968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776331.159491:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776331.159493:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776331.159495:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776331.159499:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18098 00000100:00100000:3.0:1775776331.159499:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899456:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776331.159505:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776331.159510:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776331.159515:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776331.159523:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664900096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776331.159530:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159542:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b5f80 x1862032664899456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159551:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664899456/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159562:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776331.159566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776331.159576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159582:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b4700 x1862032664899584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159589:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664899584/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159596:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159597:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776331.159601:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899712:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776331.159611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159616:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b5500 x1862032664899712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664899712/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899712:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776331.159639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776331.159649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159654:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b4a80 x1862032664899840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664899840/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159669:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664899840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776331.159672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776331.159681:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b6d80 x1862032664899968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664899968/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159710:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159713:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664899968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776331.159734:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776331.159761:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776331.159772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776331/real 1775776331] req@ffff9829764b5180 x1862032664900096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776331.159780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664900096/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776331.159786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776331.159789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776331.159790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776331.671170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776331.927237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776332.695175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776332.951293:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776333.719262:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776333.975308:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776334.743167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776334.999282:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776335.767224:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776336.023333:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776336.279248:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279256:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776336.279260:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279270:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279281:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279286:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776336.279331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664900224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776336.279375:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279378:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279382:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279385:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279394:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776336.279405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664900352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279413:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279415:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776336.279417:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279420:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279422:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279424:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279427:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776336.279432:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664900224:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776336.279439:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664900480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279447:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279449:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776336.279450:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279453:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279455:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279457:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279461:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776336.279469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664900608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776336.279485:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279487:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279489:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279493:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776336.279495:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279498:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776336.279506:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664900736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279516:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776336.279518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279521:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279523:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776336.279526:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279530:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776336.279532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2d500 x1862032664900224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776336.279539:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664900864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776336.279563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776336.279567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776336.279569:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776336.279572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18103 00000100:00080000:0.0:1775776336.279575:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776336.279577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664900224/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776336.279581:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776336.279587:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776336.279595:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776336.279599:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664900992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776336.279603:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664900224:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776336.279621:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776336.279640:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279649:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2e300 x1862032664900352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279657:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664900352/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279667:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776336.279686:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776336.279697:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279703:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2d180 x1862032664900480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279710:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664900480/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279721:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776336.279729:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900608:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776336.279740:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279745:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2c380 x1862032664900608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664900608/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279757:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279759:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279761:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900608:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776336.279765:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900736:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776336.279774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2f480 x1862032664900736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664900736/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279792:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279796:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664900736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776336.279799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776336.279808:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279813:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2fb80 x1862032664900864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279820:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664900864/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279824:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279826:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776336.279831:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900992:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776336.279840:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776336.279845:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776336/real 1775776336] req@ffff982974b2f100 x1862032664900992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776336.279852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664900992/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776336.279857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776336.279858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776336.279860:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664900992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776336.791156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776337.047284:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776337.815141:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776338.071274:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776338.839189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776339.095232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776339.863185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776340.119292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776340.887167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776341.143200:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776341.399160:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399168:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776341.399172:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399180:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399190:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399196:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776341.399232:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664901120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399254:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399257:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776341.399258:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399261:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399264:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399267:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776341.399275:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776341.399284:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664901248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399292:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399295:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776341.399295:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664901120:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776341.399299:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399305:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399307:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399309:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776341.399312:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776341.399322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664901376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399331:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776341.399332:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399336:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399349:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399351:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776341.399355:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776341.399359:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776341.399367:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664901504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399375:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399378:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776341.399379:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399382:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399385:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399388:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776341.399392:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776341.399392:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b5c00 x1862032664901120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776341.399402:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664901632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399410:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776341.399412:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664901120/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776341.399413:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776341.399416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399424:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399429:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776341.399432:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776341.399432:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775776341.399439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776341.399441:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399445:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664901120:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776341.399450:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664901760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776341.399458:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776341.399460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776341.399461:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776341.399464:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18108 00000100:00080000:2.0:1775776341.399466:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776341.399469:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901248:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776341.399473:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776341.399477:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776341.399487:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664901888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776341.399495:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b4e00 x1862032664901248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664901248/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399528:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399530:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399533:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776341.399537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776341.399549:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b7b80 x1862032664901376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399562:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664901376/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399568:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399569:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399571:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776341.399575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901504:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776341.399586:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399592:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b7100 x1862032664901504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399599:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664901504/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399604:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399605:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399607:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901504:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776341.399611:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776341.399626:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399636:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b6a00 x1862032664901632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399656:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664901632/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399672:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664901632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776341.399678:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664901760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776341.399695:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399703:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b4000 x1862032664901760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399711:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664901760/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664901760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776341.399735:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664901888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776341.399746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776341.399752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776341/real 1775776341] req@ffff9829764b6680 x1862032664901888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776341.399759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664901888/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776341.399768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776341.399770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776341.399772:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664901888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776341.911171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776342.167239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776342.935173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776343.191241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776343.959173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776344.215246:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776344.983207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776345.239291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776346.007196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776346.263287:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776346.519220:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776346.519230:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776346.519235:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519245:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519259:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519266:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776346.519307:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664902016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776346.519332:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776346.519335:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776346.519336:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519339:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519345:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776346.519356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776346.519368:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664902144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776346.519378:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664902016:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776346.519381:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776346.519387:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776346.519389:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519394:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519398:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519402:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776346.519407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776346.519417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664902272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776346.519424:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776346.519426:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776346.519427:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519430:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519433:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519436:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776346.519439:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776346.519446:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664902400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776346.519457:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776346.519459:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776346.519460:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776346.519463:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519466:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519468:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519470:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776346.519473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776346.519481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664902528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776346.519487:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776346.519489:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776346.519491:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776346.519493:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519495:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519500:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe3b80 x1862032664902016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776346.519505:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776346.519518:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664902656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776346.519522:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664902016/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519532:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776346.519535:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776346.519537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776346.519541:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776346.519544:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776346.519544:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664902016:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776346.519552:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18114 00000100:00080000:1.0:1775776346.519555:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776346.519557:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776346.519560:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776346.519567:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664902784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776346.519574:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776346.519599:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe1f80 x1862032664902144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664902144/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519639:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776346.519647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776346.519664:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe0a80 x1862032664902272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664902272/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519703:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776346.519716:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902400:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776346.519737:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519748:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe0e00 x1862032664902400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664902400/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519765:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519769:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902400:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776346.519778:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776346.519799:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe1c00 x1862032664902528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664902528/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519829:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519832:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664902528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776346.519837:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664902656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776346.519857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519867:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe1500 x1862032664902656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519879:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664902656/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519884:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519886:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519887:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664902656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776346.519901:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664902784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776346.519915:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776346.519921:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776346/real 1775776346] req@ffff982981fe1180 x1862032664902784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776346.519928:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664902784/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776346.519933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776346.519935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776346.519938:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664902784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776347.031187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776347.287253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776348.055201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776348.311249:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776349.079138:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776349.335232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776350.103203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776350.359237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776351.127166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776351.383202:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776351.639208:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639216:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776351.639220:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639229:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639239:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639245:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776351.639278:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664902912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776351.639301:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639303:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776351.639305:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639308:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639311:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639315:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639323:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776351.639331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664902912:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776351.639333:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664903040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776351.639346:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639348:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776351.639350:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639352:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639355:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639357:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639360:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776351.639369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664903168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776351.639376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639378:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776351.639380:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639382:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639386:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639388:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639392:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776351.639396:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776351.639404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664903296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776351.639411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639414:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776351.639416:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639419:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639422:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639424:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639427:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776351.639429:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9ea00 x1862032664902912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776351.639436:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664903424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776351.639445:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776351.639448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776351.639449:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776351.639453:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664902912/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776351.639455:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639460:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639466:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639472:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776351.639477:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775776351.639481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664903552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776351.639486:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776351.639493:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776351.639496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664902912:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776351.639497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776351.639500:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776351.639507:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18119 00000100:00080000:3.0:1775776351.639509:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776351.639512:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776351.639515:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776351.639521:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664903680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776351.639527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776351.639562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639577:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9e680 x1862032664903040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639592:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664903040/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639606:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639610:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776351.639617:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776351.639636:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9ed80 x1862032664903168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664903168/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639658:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639661:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639663:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776351.639666:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903296:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776351.639678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9df80 x1862032664903296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639691:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664903296/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639699:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903296:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776351.639703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776351.639712:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9f480 x1862032664903424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639725:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664903424/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639734:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776351.639737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664903552:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776351.639747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9d180 x1862032664903552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664903552/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639767:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664903552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776351.639780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664903680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776351.639790:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776351.639796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776351/real 1775776351] req@ffff982980f9c000 x1862032664903680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776351.639802:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664903680/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776351.639808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776351.639810:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776351.639812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664903680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776352.151148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776352.407242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776353.175223:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776353.431264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776354.199130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776354.455246:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776355.223174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776355.479245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776356.247219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776356.503270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776356.759186:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776356.759196:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776356.759201:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759210:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00080000:0.0:1775776356.759224:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759232:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776356.759274:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664903808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776356.759308:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776356.759312:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776356.759313:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759319:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00080000:0.0:1775776356.759324:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776356.759341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776356.759351:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664903808:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776356.759358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664903936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776356.759371:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776356.759375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776356.759377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759381:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00080000:0.0:1775776356.759385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759390:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776356.759395:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776356.759408:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664904064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776356.759418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776356.759422:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776356.759424:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759428:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18124 00000800:00000100:1.0:1775776356.759433:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776356.759435:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759441:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776356.759448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776356.759463:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664904192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776356.759471:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2ca80 x1862032664903808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776356.759480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776356.759487:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776356.759489:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776356.759493:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664903808/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776356.759498:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00080000:0.0:1775776356.759506:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776356.759516:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776356.759519:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775776356.759522:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759527:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664903808:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776356.759537:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664904320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776356.759549:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776356.759552:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903936:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776356.759555:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776356.759560:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759565:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00080000:0.0:1775776356.759570:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759574:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776356.759580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776356.759589:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776356.759594:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664904448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776356.759605:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776356.759607:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2df80 x1862032664903936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776356.759613:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776356.759618:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776356.759624:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18124 00000100:00100000:1.0:1775776356.759635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664903936/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776356.759639:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776356.759649:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776356.759652:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776356.759657:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776356.759659:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664903936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776356.759673:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776356.759678:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664904576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776356.759702:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776356.759715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2e680 x1862032664904064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776356.759729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664904064/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776356.759737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776356.759739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776356.759747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904192:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776356.759786:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776356.759798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2f100 x1862032664904192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776356.759811:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664904192/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776356.759820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776356.759823:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759828:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904192:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776356.759843:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776356.759860:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776356.759868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2fb80 x1862032664904320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776356.759876:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664904320/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776356.759881:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776356.759883:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759886:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776356.759901:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664904448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776356.759922:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776356.759934:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2f480 x1862032664904448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776356.759946:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664904448/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776356.759951:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776356.759954:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.759956:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664904448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776356.759960:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664904576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776356.759985:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776356.759993:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776356/real 1775776356] req@ffff982974b2c380 x1862032664904576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776356.760001:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664904576/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776356.760006:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776356.760008:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776356.760010:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664904576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776357.271174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776357.527280:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776358.295205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776358.551255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776359.319211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776359.575267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776360.343210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776360.599234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776361.367193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776361.623269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776361.879176:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879189:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776361.879194:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879206:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879222:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879230:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776361.879276:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664904704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776361.879306:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879311:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776361.879313:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879318:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879322:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879326:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776361.879337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776361.879350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664904832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776361.879356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664904704:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776361.879365:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879369:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776361.879370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879375:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879379:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879382:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776361.879387:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776361.879399:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664904960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776361.879406:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879410:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776361.879411:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879415:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879419:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879424:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776361.879430:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776361.879441:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664905088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776361.879451:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879455:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776361.879457:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879462:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879466:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879470:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776361.879474:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904832:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775776361.879476:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776361.879490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664905216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776361.879497:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879500:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776361.879501:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776361.879504:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879509:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776361.879514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904960:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776361.879515:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776361.879524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776361.879537:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:0.0:1775776361.879539:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664905344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776361.879550:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776361.879554:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776361.879556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776361.879559:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905216:192.168.201.118@tcp:8:kworker.0 00000100:00080000:0.0:1775776361.879563:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18129 00000100:00080000:0.0:1775776361.879567:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776361.879570:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776361.879575:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776361.879577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664905344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776361.879590:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664905472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776361.879627:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664905472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776362.391169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776362.647209:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776363.415395:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776363.671240:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776364.439192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776364.695301:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775776364.951337:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775776364.951391:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775776364.951413:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776364.951454:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776364.951464:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776364.951477:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776364.951492:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775776364.951501:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2e300 x1862032664904832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775776364.951504:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776364.951516:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775776364.951537:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664904832/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951555:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951566:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776364.951574:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2d500 x1862032664904960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951585:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664904960/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951593:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664904960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776364.951608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2c380 x1862032664905088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951622:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664905088/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951669:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905088:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776364.951676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2d180 x1862032664904704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951688:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664904704/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951695:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951701:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664904704:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776364.951731:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2f100 x1862032664905472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664905472/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951752:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951755:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664905472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776364.951765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2f480 x1862032664905216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664905216/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776364.951798:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776361/real 1775776364] req@ffff982974b2fb80 x1862032664905344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776364.951809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664905344/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776364.951818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776364.951822:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776364.951825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664905344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776365.463140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776365.719255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776366.487197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776366.743283:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776366.999173:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999182:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776366.999186:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999196:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999208:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776366.999213:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776366.999243:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664905600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776366.999277:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999280:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776366.999281:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999286:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776366.999289:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999297:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776366.999306:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664905728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776366.999313:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999315:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776366.999316:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999321:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00100000:1.0:1775776366.999321:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664905600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776366.999325:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776366.999330:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999333:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776366.999350:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664905856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776366.999359:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999362:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776366.999364:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999367:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999372:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776366.999377:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776366.999382:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999390:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776366.999404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664905984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776366.999414:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776366.999417:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2e680 x1862032664905600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776366.999419:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776366.999422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999436:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776366.999440:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776366.999453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664906112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776366.999462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999466:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776366.999468:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999472:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999477:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776366.999477:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664905600/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776366.999485:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999492:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776366.999495:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999501:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776366.999505:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664906240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776366.999508:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664905600:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776366.999520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776366.999524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776366.999525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776366.999530:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18134 00000100:00080000:0.0:1775776366.999534:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776366.999535:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905728:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776366.999544:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776366.999551:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776366.999560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664906368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776366.999586:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999630:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2df80 x1862032664905728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664905728/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999651:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999654:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776366.999660:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776366.999679:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999689:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2ca80 x1862032664905856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664905856/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999714:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776366.999722:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905984:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776366.999747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999756:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2fb80 x1862032664905984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664905984/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999776:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664905984:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776366.999783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776366.999800:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2f480 x1862032664906112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999817:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664906112/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999822:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776366.999830:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664906240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776366.999847:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999856:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2f100 x1862032664906240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999867:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664906240/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664906240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776366.999897:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664906368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776366.999915:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776366.999925:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776366/real 1775776366] req@ffff982974b2d180 x1862032664906368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776366.999935:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664906368/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776366.999942:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776366.999945:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776366.999947:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664906368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776367.511178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776367.767248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776368.535210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776368.791324:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776369.559226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776369.815266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776370.583203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776370.839268:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776371.607177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776371.863233:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776372.119216:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119230:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776372.119237:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119247:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119261:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119267:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776372.119311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664906496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776372.119335:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119338:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776372.119340:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119343:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119345:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119348:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119357:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776372.119369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664906624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776372.119386:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664906496:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776372.119402:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119405:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776372.119407:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119411:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119414:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119417:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119421:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776372.119432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664906752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776372.119438:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119440:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776372.119441:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119443:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119445:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119447:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119450:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776372.119457:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664906880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776372.119462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776372.119470:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776372.119471:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776372.119474:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119478:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119480:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776372.119494:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664907008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776372.119517:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119520:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776372.119522:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776372.119526:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe1180 x1862032664906496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776372.119529:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00080000:1.0:1775776372.119534:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119542:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119548:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776372.119555:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664906496/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776372.119562:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664907136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776372.119569:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776372.119572:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776372.119576:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776372.119576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776372.119577:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776372.119583:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18139 00000100:00100000:3.0:1775776372.119585:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664906496:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776372.119587:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776372.119592:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776372.119596:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776372.119604:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664907264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776372.119617:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776372.119643:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe1500 x1862032664906624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119665:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664906624/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119673:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119679:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776372.119683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776372.119699:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119707:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe1c00 x1862032664906752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119717:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664906752/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119728:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119731:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776372.119736:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906880:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776372.119752:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe0e00 x1862032664906880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664906880/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664906880:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776372.119792:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776372.119806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe0a80 x1862032664907008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119823:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664907008/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119830:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776372.119840:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664907136:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776372.119855:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119863:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe1f80 x1862032664907136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119874:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664907136/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664907136:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776372.119910:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664907264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776372.119928:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776372.119937:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776372/real 1775776372] req@ffff982981fe3b80 x1862032664907264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776372.119947:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664907264/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776372.119954:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776372.119957:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776372.119959:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664907264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776372.631175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776372.887279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776373.655165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776373.911295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776374.679220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776374.935254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776375.703156:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776375.959248:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776376.727215:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776376.983257:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776377.239232:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239239:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776377.239243:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239251:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239260:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776377.239265:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776377.239297:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664907392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776377.239320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776377.239324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239327:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239329:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776377.239332:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239338:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776377.239356:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664907520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776377.239363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239365:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776377.239367:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239371:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00100000:3.0:1775776377.239372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664907392:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776377.239373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776377.239378:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239384:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776377.239394:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664907648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776377.239400:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239402:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776377.239403:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239405:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239408:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776377.239420:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239424:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776377.239433:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664907776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776377.239439:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239442:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776377.239443:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239445:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239448:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776377.239451:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776377.239454:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239456:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776377.239466:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664907904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776377.239475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239478:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776377.239480:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776377.239484:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239488:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776377.239492:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2c380 x1862032664907392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776377.239493:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239500:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776377.239516:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664908032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776377.239519:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664907392/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776377.239532:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776377.239537:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776377.239538:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776377.239539:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776377.239544:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776377.239546:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18144 00000100:00080000:0.0:1775776377.239552:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776377.239552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664907392:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776377.239561:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776377.239567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776377.239578:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776377.239580:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664908160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776377.239610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2d500 x1862032664907520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239634:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664907520/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239643:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239646:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776377.239653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907648:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776377.239668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2e300 x1862032664907648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239687:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664907648/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239694:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239697:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907648:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776377.239705:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907776:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776377.239720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2d180 x1862032664907776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239738:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664907776/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907776:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776377.239756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776377.239769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2f100 x1862032664907904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664907904/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664907904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776377.239805:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908032:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776377.239820:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239829:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2f480 x1862032664908032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664908032/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908032:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776377.239869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776377.239897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776377.239906:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776377/real 1775776377] req@ffff982974b2fb80 x1862032664908160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776377.239916:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664908160/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776377.239924:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776377.239927:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776377.239929:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776377.751204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776378.007219:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776378.775187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776379.031263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776379.799180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776380.055214:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776380.823193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776381.079241:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776381.847168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776382.103244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776382.359123:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359130:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776382.359134:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359141:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359150:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359155:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776382.359184:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664908288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776382.359205:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359207:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776382.359209:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359211:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359213:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359215:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776382.359222:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776382.359230:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664908416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776382.359245:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359248:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776382.359250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359255:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00100000:3.0:1775776382.359258:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664908288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776382.359260:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359267:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776382.359272:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776382.359286:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664908544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776382.359293:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359296:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776382.359297:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359299:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359302:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359304:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776382.359309:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776382.359316:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664908672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776382.359337:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776382.359344:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359347:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776382.359349:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359352:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359356:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359359:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776382.359363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776382.359371:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2ca80 x1862032664908288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776382.359372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664908800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776382.359394:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359397:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776382.359398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776382.359401:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359402:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776382.359405:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359408:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664908288/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776382.359409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776382.359418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664908928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776382.359424:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776382.359426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776382.359430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776382.359431:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776382.359433:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776382.359436:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18149 00000100:00080000:0.0:1775776382.359439:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776382.359442:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664908288:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776382.359443:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776382.359450:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776382.359455:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776382.359457:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664909056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776382.359480:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359489:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2df80 x1862032664908416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664908416/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359503:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359505:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776382.359522:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776382.359532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359537:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2e680 x1862032664908544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359543:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664908544/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359548:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359550:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359552:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776382.359555:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908672:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776382.359565:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359569:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2fb80 x1862032664908672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664908672/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359587:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908672:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776382.359595:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776382.359612:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359623:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2f480 x1862032664908800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359635:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664908800/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359642:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359645:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359648:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664908800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776382.359654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908928:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776382.359670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359679:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2f100 x1862032664908928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359689:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664908928/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359696:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664908928:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776382.359707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776382.359719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776382.359727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776382/real 1775776382] req@ffff982974b2d180 x1862032664909056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776382.359737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664909056/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776382.359745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776382.359748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776382.359751:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776382.871168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776383.127236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776383.895170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776384.151280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776384.919166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776385.175232:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776385.943171:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776386.199213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776386.967162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776387.223215:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776387.479190:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776387.479204:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479211:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479221:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479227:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776387.479268:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664909184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479288:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479291:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776387.479292:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479295:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479298:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479300:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776387.479307:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776387.479315:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664909312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479321:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776387.479324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479326:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479327:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479329:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776387.479332:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776387.479337:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664909184:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776387.479339:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664909440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479353:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479355:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776387.479357:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479359:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479361:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479362:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776387.479366:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776387.479373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664909568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479380:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776387.479381:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479384:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479386:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479388:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776387.479391:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776387.479395:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776387.479400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664909696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479405:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479408:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776387.479408:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479410:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479419:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479422:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776387.479426:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776387.479428:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9ce00 x1862032664909184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776387.479434:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664909824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776387.479443:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776387.479445:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775776387.479446:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664909184/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776387.479447:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776387.479453:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18154 00000100:00080000:3.0:1775776387.479457:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776387.479461:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776387.479464:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775776387.479464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776387.479471:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776387.479474:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664909952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776387.479475:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664909184:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776387.479486:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776387.479502:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479512:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9e300 x1862032664909312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479520:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664909312/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479526:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479529:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479533:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776387.479556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776387.479575:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9f100 x1862032664909440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479598:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664909440/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479606:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776387.479618:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909568:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776387.479638:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9c000 x1862032664909568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479660:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664909568/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479674:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909568:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776387.479680:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776387.479698:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479709:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9d180 x1862032664909696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479720:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664909696/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479728:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479731:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479733:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664909696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776387.479739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909824:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776387.479759:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479770:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9f480 x1862032664909824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479782:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664909824/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479792:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909824:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776387.479799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776387.479813:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776387.479819:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776387/real 1775776387] req@ffff982980f9df80 x1862032664909952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776387.479826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664909952/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776387.479831:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776387.479833:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776387.479835:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664909952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776387.991170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776388.247392:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776389.015143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776389.271215:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776390.039168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776390.295254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776391.063151:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776391.319208:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776392.087220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776392.343241:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776392.599189:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776392.599199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776392.599203:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599213:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599226:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776392.599233:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776392.599274:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664910080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776392.599298:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776392.599302:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776392.599304:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599308:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599312:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776392.599316:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776392.599324:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776392.599326:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664910080:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776392.599338:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664910208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776392.599350:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776392.599353:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776392.599356:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599359:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599363:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776392.599366:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776392.599371:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776392.599381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664910336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776392.599389:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776392.599392:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776392.599393:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776392.599396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599400:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599402:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776392.599405:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776392.599409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776392.599417:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664910464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776392.599423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776392.599425:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe3480 x1862032664910080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776392.599426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776392.599428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599435:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599439:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776392.599445:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664910080/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776392.599455:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776392.599456:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775776392.599460:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776392.599468:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599472:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664910080:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776392.599476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664910592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776392.599485:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776392.599488:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776392.599491:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776392.599493:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910208:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776392.599498:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:1.0:1775776392.599502:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776392.599506:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776392.599511:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776392.599516:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776392.599523:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664910720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776392.599531:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe1880 x1862032664910208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776392.599537:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776392.599550:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664910208/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776392.599551:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776392.599554:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776392.599563:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18160 00000100:00080000:3.0:1775776392.599565:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776392.599568:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776392.599570:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599576:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910208:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776392.599577:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910336:192.168.201.118@tcp:38:kworker.0 00000008:00100000:1.0:1775776392.599584:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776392.599600:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664910848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776392.599605:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599617:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe2300 x1862032664910336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776392.599625:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664910336/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776392.599630:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776392.599633:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776392.599642:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910464:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776392.599652:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599657:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe2680 x1862032664910464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776392.599664:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664910464/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776392.599668:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776392.599669:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599671:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910464:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776392.599674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776392.599682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe2a00 x1862032664910592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776392.599693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664910592/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776392.599697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776392.599699:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664910592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776392.599711:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664910720:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776392.599728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe0380 x1862032664910720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776392.599741:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664910720/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776392.599745:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776392.599747:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599749:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664910720:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776392.599752:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664910848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776392.599760:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776392.599765:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776392/real 1775776392] req@ffff982981fe0000 x1862032664910848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776392.599771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664910848/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776392.599775:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776392.599776:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776392.599778:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664910848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776393.111155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776393.367211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776394.135181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776394.391239:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776395.159145:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776395.415213:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776396.183177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776396.439232:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776397.207180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776397.463205:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776397.719154:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719161:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776397.719165:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719173:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719182:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719186:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776397.719217:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664910976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719244:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719246:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776397.719247:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719250:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719252:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719254:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719261:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776397.719269:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664911104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719275:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776397.719279:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664910976:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776397.719279:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776397.719280:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719288:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719291:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719293:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776397.719299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664911232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719308:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719310:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776397.719312:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719313:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719315:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719317:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719320:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776397.719325:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664911360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719332:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776397.719334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719336:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719338:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719340:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719342:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776397.719347:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776397.719349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664911488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719355:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776397.719358:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719361:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00080000:3.0:1775776397.719364:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719366:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719369:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776397.719373:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664911616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776397.719379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776397.719381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776397.719382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776397.719386:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18165 00000100:00100000:1.0:1775776397.719387:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9ed80 x1862032664910976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776397.719388:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776397.719392:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776397.719396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776397.719405:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664911744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776397.719417:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664910976/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719429:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719433:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719438:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664910976:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776397.719458:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776397.719479:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719490:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9e680 x1862032664911104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719500:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664911104/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719508:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719511:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719514:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776397.719519:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911232:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776397.719533:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719541:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9ea00 x1862032664911232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719552:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664911232/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719565:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719568:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719571:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776397.719575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911360:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776397.719591:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719597:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9dc00 x1862032664911360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664911360/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719610:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911360:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776397.719614:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776397.719629:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719635:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9d880 x1862032664911488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719641:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664911488/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719647:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719650:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664911488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776397.719653:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664911616:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776397.719662:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719667:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9fb80 x1862032664911616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719673:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664911616/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719677:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719680:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664911616:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776397.719683:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664911744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776397.719691:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776397.719696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776397/real 1775776397] req@ffff982980f9d500 x1862032664911744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776397.719702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664911744/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776397.719706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776397.719707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776397.719709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664911744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776398.231155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776398.487231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776399.255169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776399.511202:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776400.279159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776400.535226:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776401.303153:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776401.559186:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776402.327276:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776402.583248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776402.839185:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839197:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776402.839201:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839211:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839222:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839228:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776402.839260:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664911872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776402.839283:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839286:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776402.839287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839289:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839292:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839295:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839305:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776402.839317:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664912000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776402.839328:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776402.839332:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664911872:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776402.839332:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776402.839334:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839347:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839351:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839355:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776402.839372:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664912128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776402.839382:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776402.839387:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839392:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839396:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839399:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776402.839404:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839405:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776402.839416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664912256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776402.839423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776402.839427:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839430:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839436:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776402.839442:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9c380 x1862032664911872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776402.839447:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664912384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776402.839461:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839463:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776402.839465:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839469:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00100000:1.0:1775776402.839470:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664911872/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776402.839471:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839477:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839482:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776402.839485:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839490:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776402.839490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664912512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776402.839496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664911872:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776402.839507:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776402.839509:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776402.839511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776402.839513:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18170 00000100:00080000:3.0:1775776402.839515:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776402.839517:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776402.839520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776402.839521:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776402.839529:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664912640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776402.839543:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839555:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9c700 x1862032664912000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839567:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664912000/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839575:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839578:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776402.839596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912128:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776402.839616:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839627:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9f800 x1862032664912128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839639:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664912128/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839652:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839656:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839659:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776402.839664:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912256:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776402.839684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839693:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9d500 x1862032664912256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839700:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664912256/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839705:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839707:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912256:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776402.839713:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912384:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776402.839724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839729:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9fb80 x1862032664912384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839735:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664912384/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839739:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776402.839747:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664912512:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776402.839756:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839760:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9d880 x1862032664912512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664912512/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839771:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839773:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664912512:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776402.839787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664912640:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776402.839795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776402.839801:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776402/real 1775776402] req@ffff982980f9dc00 x1862032664912640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776402.839808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664912640/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776402.839813:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776402.839814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776402.839816:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664912640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776403.351179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776403.607242:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776404.375169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776404.631263:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776405.399148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776405.655264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776406.423204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776406.679268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776407.447165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776407.703267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776407.959175:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959182:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776407.959185:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959193:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959203:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959207:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776407.959238:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664912768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776407.959258:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959261:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776407.959262:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959265:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959267:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959270:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776407.959277:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776407.959287:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664912896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776407.959288:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664912768:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776407.959299:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959301:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776407.959302:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959305:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959306:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959308:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776407.959311:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776407.959317:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664913024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776407.959327:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959329:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776407.959330:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959332:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959334:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959336:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776407.959339:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776407.959345:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776407.959346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664913152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776407.959359:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776407.959362:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959365:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959368:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959370:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776407.959373:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776407.959374:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe0000 x1862032664912768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776407.959381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664913280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776407.959390:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776407.959393:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664912768/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776407.959394:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776407.959396:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959407:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959410:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776407.959411:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775776407.959415:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776407.959418:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959421:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664912768:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776407.959425:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664913408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776407.959431:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776407.959433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776407.959434:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776407.959436:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18175 00000100:00080000:1.0:1775776407.959438:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776407.959440:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776407.959443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776407.959445:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776407.959454:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664913536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776407.959469:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959482:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe0380 x1862032664912896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959495:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664912896/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959503:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959506:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959509:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664912896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776407.959514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913024:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776407.959528:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959535:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe2a00 x1862032664913024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959541:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664913024/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959548:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959550:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776407.959553:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913152:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776407.959562:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959568:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe2680 x1862032664913152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664913152/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959578:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959580:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959581:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913152:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776407.959585:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776407.959593:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959598:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe2300 x1862032664913280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959604:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664913280/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959608:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959609:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959611:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776407.959613:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664913408:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776407.959621:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959626:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe1880 x1862032664913408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664913408/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959638:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959640:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664913408:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776407.959654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664913536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776407.959670:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776407.959680:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776407/real 1775776407] req@ffff982981fe3480 x1862032664913536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776407.959690:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664913536/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776407.959698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776407.959701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776407.959704:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664913536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776408.471198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776408.727294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776409.495214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776409.751295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776410.519221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776410.775292:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776411.543179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776411.799311:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776412.567205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776412.823281:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776413.079233:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776413.079241:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776413.079245:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079255:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079266:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079271:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776413.079312:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664913664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776413.079345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776413.079350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776413.079352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079357:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079366:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079370:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776413.079379:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776413.079390:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664913792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776413.079399:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664913664:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776413.079401:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776413.079408:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776413.079410:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079414:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079416:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079418:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776413.079421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776413.079432:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664913920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776413.079439:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776413.079442:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776413.079443:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079445:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079447:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079450:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776413.079455:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776413.079467:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664914048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776413.079477:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776413.079485:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776413.079490:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776413.079493:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079498:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079504:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079509:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079513:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2e300 x1862032664913664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776413.079516:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776413.079534:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664914176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776413.079536:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664913664/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079566:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776413.079568:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776413.079571:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776413.079576:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775776413.079577:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664913664:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776413.079580:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079587:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079593:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776413.079596:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079600:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913792:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775776413.079601:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776413.079617:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664914304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776413.079626:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776413.079626:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776413.079631:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776413.079633:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776413.079636:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18180 00000100:00080000:0.0:1775776413.079640:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776413.079642:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2d500 x1862032664913792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776413.079645:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776413.079651:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776413.079661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664913792/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079671:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776413.079675:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664914432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776413.079676:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.079684:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776413.079698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776413.079724:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079735:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2c380 x1862032664913920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776413.079744:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664913920/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776413.079752:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.079754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664913920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776413.079759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914048:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776413.079774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2d180 x1862032664914048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776413.079809:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664914048/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079818:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776413.079821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.079825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914048:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776413.079831:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776413.079854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079865:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2f100 x1862032664914176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776413.079877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664914176/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776413.079890:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.079892:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776413.079909:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664914304:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776413.079931:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079943:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2f480 x1862032664914304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776413.079954:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664914304/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079960:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776413.079962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.079964:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664914304:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776413.079967:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664914432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776413.079980:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776413.079986:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776413/real 1775776413] req@ffff982974b2fb80 x1862032664914432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776413.079993:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664914432/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776413.079998:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776413.080000:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776413.080002:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664914432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776413.591214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776413.847299:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776414.615158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776414.871258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776415.639161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776415.895237:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776416.663167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776416.919237:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776417.687150:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776417.943235:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776418.199183:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199190:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776418.199195:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199203:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199212:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199217:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776418.199248:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664914560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199270:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199273:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776418.199274:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199277:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199278:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199281:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199288:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776418.199295:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664914688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199302:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199304:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776418.199306:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199310:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199312:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199314:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199317:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776418.199322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664914816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199335:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199341:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775776418.199343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664914560:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776418.199344:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199351:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199358:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199361:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776418.199374:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664914944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199380:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776418.199384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199387:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199390:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199395:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776418.199409:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664915072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199418:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199421:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776418.199422:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199425:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199430:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199433:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199436:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776418.199445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776418.199446:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664915200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776418.199455:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776418.199457:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776418.199459:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776418.199462:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18185 00000100:00080000:3.0:1775776418.199465:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776418.199468:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776418.199470:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776418.199476:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664915328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776418.199505:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9ea00 x1862032664914560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199525:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664914560/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199533:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199537:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199540:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664914560:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776418.199548:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776418.199574:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199587:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9e680 x1862032664914688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199599:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664914688/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199607:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776418.199631:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776418.199645:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199652:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9ed80 x1862032664914816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664914816/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199664:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199666:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199668:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776418.199671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914944:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776418.199682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199688:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9df80 x1862032664914944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199716:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664914944/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199722:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199724:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199726:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664914944:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776418.199731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776418.199746:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199753:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9f480 x1862032664915072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664915072/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199766:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199768:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776418.199771:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664915200:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776418.199780:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9d180 x1862032664915200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664915200/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199796:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199800:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664915200:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776418.199803:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664915328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776418.199812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776418.199817:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776418/real 1775776418] req@ffff982980f9c000 x1862032664915328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776418.199822:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664915328/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776418.199826:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776418.199828:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776418.199830:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664915328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776418.711197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776418.967188:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776419.735155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776419.991285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776420.759174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776421.015198:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776421.783191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776422.039256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776422.807176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776423.063256:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776423.319162:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319174:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776423.319178:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319195:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319209:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319215:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776423.319248:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664915456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776423.319275:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319278:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776423.319280:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319284:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319287:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319291:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319299:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776423.319311:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664915584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776423.319320:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776423.319325:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319335:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319340:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776423.319351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664915712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776423.319360:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664915456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776423.319364:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319368:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776423.319370:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319375:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319379:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319383:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319388:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776423.319400:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664915840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776423.319409:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319413:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776423.319415:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319418:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319422:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775776423.319426:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776423.319430:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776423.319446:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664915968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776423.319456:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776423.319459:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776423.319462:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776423.319463:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2e680 x1862032664915456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776423.319470:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319476:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319483:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776423.319493:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664915456/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776423.319501:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664916096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776423.319501:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319508:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319515:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664915456:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776423.319521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776423.319525:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915584:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776423.319528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776423.319531:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776423.319535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18190 00000100:00080000:0.0:1775776423.319539:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776423.319542:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776423.319547:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776423.319552:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776423.319560:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664916224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776423.319583:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2df80 x1862032664915584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319598:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664915584/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776423.319745:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776423.319764:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776423.319772:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2ca80 x1862032664915712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319780:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664915712/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319789:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319791:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319793:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776423.319797:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915840:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776423.319807:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776423.319812:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2fb80 x1862032664915840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664915840/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319825:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915840:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776423.319829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776423.319837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776423.319842:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2f480 x1862032664915968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319848:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664915968/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319855:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664915968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776423.319864:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916096:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776423.319892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776423.319902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2f100 x1862032664916096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664916096/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319932:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319935:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319937:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916096:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776423.319940:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776423.319954:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776423.319960:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776423/real 1775776423] req@ffff982974b2d180 x1862032664916224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776423.319967:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664916224/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776423.319971:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776423.319973:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776423.319974:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776423.831173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776424.087259:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776424.855176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776425.112694:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776425.879210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776426.135247:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776426.903165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776427.159237:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776427.927154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776428.183238:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776428.439136:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439155:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776428.439162:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439173:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439185:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439191:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776428.439232:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664916352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439259:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439267:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776428.439269:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439274:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439277:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439282:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439291:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776428.439313:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664916352:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776428.439317:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664916480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439330:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439333:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776428.439335:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439339:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439343:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439347:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439352:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776428.439364:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664916608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439381:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439385:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776428.439386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439391:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439395:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439399:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776428.439417:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776428.439423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664916736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439435:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439438:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776428.439440:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439444:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439449:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776428.439452:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916608:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776428.439456:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439463:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776428.439474:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776428.439477:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664916864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439489:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439492:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775776428.439495:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916864:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776428.439496:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439504:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439509:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439513:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439529:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776428.439541:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664916992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776428.439560:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776428.439563:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776428.439565:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776428.439568:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18195 00000100:00080000:3.0:1775776428.439572:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776428.439575:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776428.439580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776428.439589:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664917120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776428.439593:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776428.439638:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664917120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776428.951174:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776429.207224:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776429.975162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776430.231262:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776430.999175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776431.255250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775776431.511814:0:404317:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775776431.511862:0:404317:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775776431.511879:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511921:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511930:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511937:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511944:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511951:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776431.511958:0:404317:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776431.511992:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9df80 x1862032664917120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512025:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664917120/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512038:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512043:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512047:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664917120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776431.512054:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9f480 x1862032664916992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512087:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664916992/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512097:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512101:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512105:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664916992:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776431.512113:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9d180 x1862032664916864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512129:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664916864/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512137:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512140:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512143:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776431.512150:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9c000 x1862032664916736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512161:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664916736/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512168:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512171:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512174:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916736:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776431.512181:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9e300 x1862032664916480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512193:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664916480/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512200:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512204:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512208:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776431.512220:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9ce00 x1862032664916608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512233:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664916608/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512241:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512244:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512247:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664916608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776431.512254:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776428/real 1775776431] req@ffff982980f9f100 x1862032664916352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776431.512267:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664916352/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776431.512278:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776431.512281:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776431.512285:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664916352:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776432.023140:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776432.279232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776433.047167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776433.303257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776433.559220:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559238:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776433.559242:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776433.559252:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559263:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559270:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776433.559304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664917248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776433.559336:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559339:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776433.559341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776433.559345:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559349:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559353:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776433.559363:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776433.559375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664917376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776433.559385:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559387:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776433.559389:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776433.559393:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559396:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559399:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776433.559404:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776433.559414:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664917504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776433.559416:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664917248:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776433.559423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559454:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776433.559455:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776433.559459:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559462:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559465:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776433.559470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776433.559480:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664917632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776433.559488:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559491:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776433.559492:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776433.559492:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776433.559495:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559498:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559501:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776433.559505:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776433.559513:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664917760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776433.559520:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559523:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776433.559525:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776433.559527:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2c380 x1862032664917248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776433.559528:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00080000:0.0:1775776433.559531:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559535:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776433.559540:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776433.559549:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664917248/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776433.559552:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664917888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776433.559560:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776433.559561:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776433.559564:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776433.559565:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776433.559566:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776433.559569:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18201 00000100:00100000:3.0:1775776433.559569:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664917248:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776433.559572:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776433.559576:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559577:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917376:192.168.201.118@tcp:38:kworker.0 00000008:00100000:0.0:1775776433.559581:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776433.559589:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664918016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776433.559599:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559610:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2d500 x1862032664917376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.559621:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664917376/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.559629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.559632:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.559634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776433.559659:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776433.559677:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2e300 x1862032664917504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.559698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664917504/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.559706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.559709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.559712:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776433.559717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917632:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776433.559736:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2d180 x1862032664917632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.559757:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664917632/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.559764:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.559767:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.559771:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917632:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776433.559776:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776433.559793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559803:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2f100 x1862032664917760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.559818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664917760/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.559826:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.559831:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.559834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664917760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776433.559838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664917888:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776433.559854:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.559868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2f480 x1862032664917888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.559878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664917888/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.559886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.559889:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.559971:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664917888:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776433.559978:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776433.560001:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776433.560016:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776433/real 1775776433] req@ffff982974b2fb80 x1862032664918016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776433.560025:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664918016/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776433.560031:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776433.560034:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776433.560037:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776434.071146:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776434.327222:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776435.095159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776435.351242:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776436.119142:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776436.375194:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776437.143179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776437.399234:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776438.167194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776438.423245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776438.679209:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679218:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776438.679223:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679233:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679247:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679253:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776438.679293:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664918144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679322:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679325:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776438.679326:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679329:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679333:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679360:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776438.679371:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664918272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679379:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679382:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775776438.679383:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664918144:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776438.679384:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679388:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679393:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679395:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679398:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776438.679405:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664918400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679411:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679413:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776438.679414:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679416:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679418:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679420:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679426:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776438.679432:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664918528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776438.679441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679443:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679446:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679448:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776438.679456:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664918656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679463:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776438.679464:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775776438.679469:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776438.679469:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679471:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679474:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679476:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776438.679482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664918784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776438.679489:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776438.679492:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776438.679494:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776438.679497:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18206 00000100:00080000:3.0:1775776438.679499:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776438.679502:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776438.679505:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776438.679510:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664918912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776438.679520:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9f100 x1862032664918144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679541:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664918144/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679552:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679557:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679560:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664918144:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776438.679587:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776438.679610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9ce00 x1862032664918272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679631:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664918272/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679644:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679648:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776438.679656:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918400:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776438.679676:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679686:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9e300 x1862032664918400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679698:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664918400/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679706:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679709:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679711:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918400:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776438.679717:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918528:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776438.679738:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9c000 x1862032664918528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679767:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664918528/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679774:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679777:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679781:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918528:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776438.679786:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776438.679806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679816:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9d180 x1862032664918656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679827:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664918656/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679835:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679837:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679841:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664918656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776438.679846:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918784:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776438.679864:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9f480 x1862032664918784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679886:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664918784/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679894:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679897:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679899:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918784:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776438.679905:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776438.679922:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776438.679932:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776438/real 1775776438] req@ffff982980f9df80 x1862032664918912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776438.679942:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664918912/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776438.679950:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776438.679953:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776438.679956:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664918912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776439.191202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776439.447253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776440.215183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776440.471198:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776441.239168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776441.495236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776442.263177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776442.519211:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776443.287195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776443.543229:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776443.799200:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799210:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776443.799214:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799224:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799236:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776443.799306:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664919040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776443.799335:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799338:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776443.799339:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799343:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799346:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799349:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799359:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776443.799372:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664919168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776443.799383:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776443.799388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799394:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799396:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776443.799406:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664919296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776443.799413:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799416:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776443.799417:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776443.799429:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664919040:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776443.799429:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799434:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799441:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799450:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776443.799457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664919424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776443.799464:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799467:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776443.799469:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799473:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799476:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799479:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799482:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776443.799489:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664919552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776443.799494:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776443.799498:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799501:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799503:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799506:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799509:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776443.799516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664919680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776443.799527:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776443.799531:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776443.799534:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776443.799537:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776443.799541:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18211 00000100:00080000:3.0:1775776443.799552:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776443.799556:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776443.799563:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776443.799573:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664919808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776443.799575:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9ed80 x1862032664919040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799660:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664919040/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799672:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799681:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799686:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664919040:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776443.799694:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776443.799716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9e680 x1862032664919168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664919168/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799739:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799741:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776443.799760:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919296:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776443.799770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799775:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9ea00 x1862032664919296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799782:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664919296/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799786:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776443.799793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919424:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776443.799803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799808:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9dc00 x1862032664919424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799818:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664919424/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799823:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799824:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799826:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919424:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776443.799829:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776443.799838:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799843:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9d880 x1862032664919552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799848:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664919552/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799852:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799854:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799855:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664919552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776443.799858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664919680:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776443.799867:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9fb80 x1862032664919680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799877:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664919680/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799888:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664919680:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776443.799892:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664919808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776443.799901:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776443.799905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776443/real 1775776443] req@ffff982980f9d500 x1862032664919808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776443.799910:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664919808/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776443.799915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776443.799917:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776443.799918:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664919808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776444.311195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776444.567289:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776445.335176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776445.591267:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776446.359183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776446.615251:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776447.383162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776447.639258:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776448.407130:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776448.663216:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776448.919182:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919189:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776448.919192:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919200:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919209:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919214:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776448.919247:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664919936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919273:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919277:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776448.919279:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919283:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919287:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919291:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919300:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776448.919311:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664920064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919318:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919322:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775776448.919324:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664919936:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776448.919324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919329:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919332:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919337:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776448.919344:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664920192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919350:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919352:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776448.919353:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919357:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919358:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919361:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919364:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776448.919369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664920320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919374:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919377:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776448.919378:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919382:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919384:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919387:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776448.919388:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776448.919394:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664920448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776448.919401:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776448.919403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919405:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919408:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919410:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919413:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776448.919419:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664920576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776448.919426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776448.919427:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b6680 x1862032664919936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776448.919430:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776448.919432:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776448.919438:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18216 00000100:00080000:2.0:1775776448.919442:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776448.919447:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776448.919452:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776448.919461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664920704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776448.919470:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664919936/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919481:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919486:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919491:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664919936:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776448.919513:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920064:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776448.919536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919548:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b4000 x1862032664920064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919560:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664920064/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919567:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919571:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919574:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776448.919580:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920192:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776448.919597:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919608:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b6a00 x1862032664920192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664920192/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919636:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920192:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776448.919645:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920320:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776448.919665:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919674:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b7100 x1862032664920320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664920320/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919694:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919697:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920320:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776448.919702:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776448.919720:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b7b80 x1862032664920448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664920448/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919748:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919750:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776448.919759:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664920576:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776448.919775:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b4e00 x1862032664920576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664920576/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919811:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919814:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664920576:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776448.919823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664920704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776448.919841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776448.919850:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776448/real 1775776448] req@ffff9829764b5c00 x1862032664920704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776448.919859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664920704/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776448.919867:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776448.919870:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776448.919872:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664920704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776449.431226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776449.687247:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776450.455178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776450.711269:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776451.479166:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776451.735257:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776452.503211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776452.759265:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776453.527158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776453.783261:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776454.039219:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039231:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776454.039236:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039246:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039260:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039265:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776454.039300:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664920832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039324:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039327:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776454.039329:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039332:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039334:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039337:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776454.039345:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776454.039354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664920960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039361:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039364:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776454.039365:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039371:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00100000:1.0:1775776454.039372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664920832:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776454.039373:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039378:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776454.039383:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776454.039393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664921088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039400:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776454.039405:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039407:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039410:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039413:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776454.039416:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776454.039423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664921216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039429:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039431:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776454.039433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775776454.039438:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776454.039441:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039445:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039448:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776454.039452:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776454.039460:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664921344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039466:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039468:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776454.039470:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039472:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039475:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039477:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776454.039481:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776454.039482:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2ca80 x1862032664920832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776454.039490:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664921472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776454.039502:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776454.039505:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775776454.039507:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664920832/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776454.039508:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776454.039515:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18221 00000100:00080000:0.0:1775776454.039520:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776454.039526:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776454.039529:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776454.039530:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775776454.039538:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:0.0:1775776454.039542:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664921600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776454.039545:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664920832:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776454.039558:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776454.039578:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039587:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2df80 x1862032664920960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039596:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664920960/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039602:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039604:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039607:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664920960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776454.039623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921088:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776454.039634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039641:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2e680 x1862032664921088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039648:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664921088/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039654:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039656:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776454.039661:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921216:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776454.039680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2fb80 x1862032664921216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039694:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664921216/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039699:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921216:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776454.039706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776454.039716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039721:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2f480 x1862032664921344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664921344/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039734:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039736:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039738:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776454.039741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664921472:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776454.039758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2f100 x1862032664921472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039770:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664921472/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039779:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664921472:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776454.039783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664921600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776454.039793:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776454.039799:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776454/real 1775776454] req@ffff982974b2d180 x1862032664921600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776454.039805:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664921600/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776454.039810:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776454.039812:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776454.039815:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664921600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776454.551191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776454.807287:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776455.575173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776455.831291:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776456.599179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776456.855245:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776457.623135:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776457.879250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776458.647173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776458.903250:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776459.159208:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159219:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776459.159224:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159234:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159248:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776459.159255:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776459.159290:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664921728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776459.159319:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159322:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776459.159325:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159329:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159332:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776459.159335:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776459.159343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776459.159355:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664921856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776459.159367:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776459.159372:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664921728:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776459.159373:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776459.159379:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159385:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159389:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776459.159394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776459.159400:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776459.159415:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664921984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776459.159426:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159428:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776459.159430:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159432:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159436:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776459.159439:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776459.159442:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776459.159448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776459.159459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664922112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776459.159466:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159469:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776459.159470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159476:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776459.159479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2e300 x1862032664921728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776459.159482:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776459.159488:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776459.159500:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664922240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776459.159506:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664921728/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776459.159514:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159522:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776459.159524:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776459.159525:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776459.159529:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159532:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776459.159532:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776459.159541:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159542:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664921728:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776459.159546:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776459.159561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664922368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776459.159566:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921856:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776459.159574:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776459.159577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776459.159579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776459.159583:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18226 00000100:00080000:0.0:1775776459.159587:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776459.159590:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776459.159594:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776459.159601:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776459.159605:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664922496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776459.159622:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2d500 x1862032664921856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159638:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664921856/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159649:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159651:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776459.159655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921984:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776459.159678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2c380 x1862032664921984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159693:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664921984/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159698:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159700:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664921984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776459.159706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922112:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776459.159718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2d180 x1862032664922112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159731:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664922112/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922112:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776459.159751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776459.159770:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159781:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2f100 x1862032664922240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159794:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664922240/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159802:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159804:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776459.159824:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664922368:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776459.159846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2f480 x1862032664922368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159871:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664922368/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159879:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664922368:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776459.159886:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664922496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776459.159900:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776459.159908:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776459/real 1775776459] req@ffff982974b2fb80 x1862032664922496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776459.159915:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664922496/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776459.159920:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776459.159922:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776459.159924:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664922496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776459.671205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776459.927262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776460.695406:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776460.951269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776461.719253:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776461.975286:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776462.743282:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776462.999308:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776463.767203:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776464.023318:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776464.279188:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279199:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776464.279206:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776464.279220:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:0.0:1775776464.279237:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776464.279247:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776464.279305:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664922624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776464.279333:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279336:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776464.279337:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776464.279341:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:0.0:1775776464.279345:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776464.279349:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776464.279359:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776464.279370:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664922752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776464.279378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776464.279382:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776464.279385:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:0.0:1775776464.279388:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776464.279394:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776464.279396:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664922624:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776464.279397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776464.279416:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664922880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776464.279423:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279426:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776464.279428:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776464.279431:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:0.0:1775776464.279433:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776464.279436:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776464.279439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776464.279448:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664923008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776464.279457:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279460:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776464.279461:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776464.279464:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18231 00000800:00000100:3.0:1775776464.279485:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776464.279500:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776464.279507:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776464.279520:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776464.279532:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2e680 x1862032664922624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776464.279533:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664923136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776464.279580:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279582:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776464.279587:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776464.279587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664922624/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776464.279606:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:3.0:1775776464.279610:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776464.279613:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776464.279616:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776464.279629:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776464.279629:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664922624:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776464.279644:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776464.279663:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664923264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776464.279664:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922752:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776464.279692:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776464.279696:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776464.279701:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776464.279701:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776464.279716:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18231 00000100:00080000:0.0:1775776464.279724:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776464.279724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2df80 x1862032664922752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776464.279737:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776464.279748:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776464.279766:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664922752/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776464.279768:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664923392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776464.279835:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.279845:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.279850:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776464.279869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776464.279927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776464.279940:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2ca80 x1862032664922880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776464.279954:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664922880/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776464.279963:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.279967:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.279971:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664922880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776464.279978:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923008:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776464.280002:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776464.280013:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2fb80 x1862032664923008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776464.280025:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664923008/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776464.280041:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.280045:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.280048:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923008:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776464.280100:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776464.280126:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776464.280137:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2f480 x1862032664923136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776464.280149:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664923136/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776464.280159:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.280163:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.280166:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776464.280187:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664923264:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776464.280209:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776464.280220:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2f100 x1862032664923264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776464.280233:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664923264/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776464.280241:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.280244:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.280248:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664923264:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776464.280255:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664923392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776464.280275:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776464.280286:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776464/real 1775776464] req@ffff982974b2d180 x1862032664923392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776464.280298:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664923392/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776464.280306:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776464.280309:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776464.280313:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664923392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776464.791190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776465.047285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776465.815193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776466.071281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776466.839198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776467.095264:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776467.863191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776468.119260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776468.887165:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776469.143267:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776469.399246:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399258:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776469.399264:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776469.399275:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399289:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399298:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776469.399339:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664923520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776469.399370:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399374:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776469.399376:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776469.399382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399387:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399392:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776469.399403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776469.399409:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664923520:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776469.399423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664923648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776469.399438:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399442:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776469.399444:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776469.399449:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399453:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399459:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776469.399465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776469.399476:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664923776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776469.399488:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776469.399490:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776469.399497:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776469.399500:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776469.399506:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399511:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399515:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776469.399522:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776469.399538:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664923904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776469.399545:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2c380 x1862032664923520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776469.399572:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399577:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776469.399579:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776469.399589:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664923520/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776469.399598:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399606:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399612:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776469.399620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775776469.399621:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776469.399628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.399634:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664923520:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776469.399640:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664924032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776469.399651:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399655:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776469.399657:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776469.399661:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00100000:3.0:1775776469.399667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923648:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776469.399676:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776469.399687:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776469.399694:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:0.0:1775776469.399698:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776469.399708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2d500 x1862032664923648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776469.399718:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664924160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776469.399727:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664923648/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776469.399738:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776469.399749:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776469.399751:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776469.399752:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776469.399771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776469.399772:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18236 00000100:00080000:0.0:1775776469.399787:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776469.399790:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923648:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776469.399796:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776469.399817:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776469.399823:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776469.399832:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664924288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776469.399856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776469.399872:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2e300 x1862032664923776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776469.399884:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664923776/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776469.399891:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776469.399894:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.399896:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776469.399901:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923904:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776469.399917:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776469.399924:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2d180 x1862032664923904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776469.399931:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664923904/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776469.399937:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776469.399940:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.399943:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664923904:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776469.399949:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924032:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776469.399968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776469.399978:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2f100 x1862032664924032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776469.399990:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664924032/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776469.399998:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776469.400001:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.400004:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776469.400023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664924160:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776469.400037:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776469.400044:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2f480 x1862032664924160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776469.400051:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664924160/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776469.400056:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776469.400059:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.400060:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664924160:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776469.400089:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664924288:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776469.400101:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776469.400108:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776469/real 1775776469] req@ffff982974b2fb80 x1862032664924288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776469.400114:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664924288/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776469.400119:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776469.400122:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776469.400123:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664924288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776469.911238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776470.167294:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776470.935147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776471.191269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776471.959179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776472.215260:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776472.983155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776473.239225:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776474.007218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776474.263292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776474.519268:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519278:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776474.519283:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519295:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519311:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519319:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776474.519368:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664924416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776474.519407:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519410:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776474.519412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519416:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519419:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519423:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519434:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776474.519449:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664924544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776474.519461:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519465:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776474.519467:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519472:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00100000:3.0:1775776474.519478:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664924416:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776474.519479:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519488:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776474.519509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664924672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776474.519522:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776474.519529:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519534:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519538:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519543:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519548:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776474.519558:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664924800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776474.519565:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519568:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776474.519569:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519573:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519576:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776474.519581:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776474.519582:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519589:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776474.519597:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664924928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776474.519604:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519607:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776474.519609:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519612:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519614:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519617:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519620:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776474.519628:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664925056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776474.519634:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2ca80 x1862032664924416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776474.519646:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776474.519653:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776474.519656:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776474.519663:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18242 00000100:00080000:0.0:1775776474.519668:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776474.519671:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776474.519674:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776474.519684:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664925184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776474.519684:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664924416/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.519737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.519748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.519754:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664924416:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776474.519764:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924544:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776474.519795:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.519814:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2df80 x1862032664924544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.519828:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664924544/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.519838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.519841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.519845:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924544:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776474.519874:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776474.519898:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.519911:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2e680 x1862032664924672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.519924:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664924672/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.519933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.519937:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.519941:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776474.519948:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924800:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776474.519968:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.519978:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2fb80 x1862032664924800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.519990:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664924800/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.519998:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.520001:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.520004:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924800:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776474.520010:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776474.520030:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.520042:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2f480 x1862032664924928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.520101:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664924928/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.520111:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.520115:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.520120:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664924928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776474.520136:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925056:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776474.520160:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.520173:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2f100 x1862032664925056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.520185:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664925056/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.520195:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.520199:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.520202:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925056:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776474.520208:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776474.520228:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776474.520239:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776474/real 1775776474] req@ffff982974b2d180 x1862032664925184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776474.520253:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664925184/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776474.520261:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776474.520265:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776474.520268:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776475.031157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776475.287230:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776476.055232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776476.311280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776477.079231:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776477.335227:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776478.103169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776478.359306:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776479.127173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776479.383272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776479.639232:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639244:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776479.639249:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639259:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639272:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639279:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776479.639320:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664925312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776479.639347:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776479.639353:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639358:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639363:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639367:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776479.639377:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776479.639391:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664925440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776479.639395:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664925312:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776479.639402:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639405:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776479.639407:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639411:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639414:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639417:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776479.639421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776479.639430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664925568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776479.639438:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639441:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776479.639443:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639446:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639449:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639452:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776479.639458:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776479.639458:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776479.639466:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664925696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776479.639475:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639478:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776479.639480:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639483:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639487:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639490:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776479.639495:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776479.639504:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664925824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776479.639512:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639516:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776479.639517:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639521:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:1.0:1775776479.639524:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776479.639528:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639528:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe3b80 x1862032664925312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776479.639533:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776479.639542:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664925952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776479.639546:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664925312/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776479.639552:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776479.639555:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776479.639557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776479.639558:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776479.639561:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18247 00000100:00080000:3.0:1775776479.639561:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776479.639565:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776479.639565:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664925312:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776479.639568:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776479.639572:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776479.639579:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664926080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776479.639588:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925440:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776479.639611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639648:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe1f80 x1862032664925440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639659:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664925440/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639665:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639667:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639669:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925440:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776479.639674:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925568:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776479.639696:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639708:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe0a80 x1862032664925568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639721:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664925568/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639736:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639744:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925568:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776479.639750:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925696:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776479.639774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639785:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe0e00 x1862032664925696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639798:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664925696/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639806:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639809:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639811:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925696:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776479.639815:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776479.639831:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639839:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe1c00 x1862032664925824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664925824/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639856:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664925824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776479.639859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925952:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776479.639869:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639874:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe1500 x1862032664925952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639881:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664925952/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639886:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639887:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639889:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664925952:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776479.639902:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776479.639913:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776479.639919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776479/real 1775776479] req@ffff982981fe1180 x1862032664926080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776479.639926:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664926080/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776479.639940:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776479.639942:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776479.639947:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776480.151175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776480.407269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776481.175196:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776481.431310:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776482.199204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776482.455305:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776483.223230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776483.479295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776484.247226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776484.503276:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776484.759229:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759243:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776484.759249:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776484.759261:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759277:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759295:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776484.759346:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664926208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776484.759380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759384:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776484.759386:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776484.759391:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759403:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759409:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776484.759422:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776484.759437:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664926336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776484.759446:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664926208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776484.759452:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759456:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776484.759458:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776484.759463:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759467:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759471:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776484.759477:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776484.759490:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664926464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776484.759499:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759503:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776484.759504:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776484.759508:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759513:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759517:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776484.759523:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776484.759528:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776484.759565:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664926592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776484.759576:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759580:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776484.759582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776484.759587:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00100000:3.0:1775776484.759591:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe1180 x1862032664926208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776484.759593:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759602:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776484.759610:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776484.759619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664926208/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776484.759624:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664926720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776484.759638:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776484.759642:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776484.759646:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776484.759648:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776484.759651:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776484.759653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664926208:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776484.759658:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759664:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759669:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776484.759674:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776484.759685:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664926848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776484.759690:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926336:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776484.759705:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776484.759709:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776484.759711:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776484.759722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776484.759727:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18252 00000100:00080000:1.0:1775776484.759732:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776484.759736:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776484.759739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe1500 x1862032664926336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776484.759743:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776484.759756:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664926336/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776484.759763:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664926976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776484.759769:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.759776:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.759782:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926336:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776484.759789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776484.759809:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776484.759820:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe1c00 x1862032664926464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776484.759830:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664926464/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776484.759838:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.759841:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.759844:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776484.759850:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926592:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776484.759868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776484.759878:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe0e00 x1862032664926592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776484.759889:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664926592/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776484.759896:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.759899:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.759902:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926592:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776484.759907:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776484.759926:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776484.759935:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe0a80 x1862032664926720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776484.759952:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664926720/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776484.759959:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.759962:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.759965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664926720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776484.759982:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926848:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776484.760001:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776484.760020:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe1f80 x1862032664926848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776484.760033:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664926848/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776484.760041:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.760044:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.760047:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926848:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776484.760053:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776484.760102:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776484.760113:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776484/real 1775776484] req@ffff982981fe3b80 x1862032664926976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776484.760124:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664926976/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776484.760139:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776484.760142:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776484.760145:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664926976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776485.271302:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776485.527296:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776486.295207:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776486.551284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776487.319212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776487.575285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776488.343164:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776488.599270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776489.367180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776489.623252:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776489.879197:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879208:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776489.879213:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776489.879223:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879251:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879259:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776489.879299:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664927104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776489.879329:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879334:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776489.879336:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776489.879340:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879344:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879350:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776489.879360:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776489.879369:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664927104:192.168.201.118@tcp:250:kworker.0 00000100:00100000:0.0:1775776489.879379:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664927232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776489.879392:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879396:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776489.879398:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776489.879403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879408:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879412:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776489.879418:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776489.879431:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664927360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776489.879443:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776489.879459:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879474:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776489.879476:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776489.879482:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879487:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879493:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776489.879499:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2e300 x1862032664927104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776489.879501:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776489.879521:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664927488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776489.879524:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664927104/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776489.879555:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879561:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776489.879563:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776489.879564:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776489.879574:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776489.879575:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879583:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776489.879584:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664927104:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776489.879592:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776489.879599:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776489.879617:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927232:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776489.879624:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664927616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776489.879638:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776489.879642:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776489.879644:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776489.879649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776489.879653:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879658:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879665:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776489.879665:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2d500 x1862032664927232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775776489.879674:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776489.879681:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664927232/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.879688:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775776489.879689:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664927744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776489.879695:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.879702:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927232:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776489.879713:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776489.879714:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927360:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776489.879719:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776489.879721:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776489.879726:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18257 00000100:00080000:0.0:1775776489.879731:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776489.879737:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776489.879743:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776489.879751:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776489.879757:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664927872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776489.879766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2c380 x1862032664927360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776489.879782:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664927360/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.879794:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776489.879798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.879802:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776489.879809:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927488:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776489.879830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776489.879839:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2d180 x1862032664927488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776489.879847:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664927488/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.879853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776489.879855:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.879857:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927488:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776489.879869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776489.879888:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776489.879899:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2f100 x1862032664927616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776489.879912:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664927616/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.879921:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776489.879924:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.879928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664927616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776489.879934:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664927744:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776489.879954:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776489.879964:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2f480 x1862032664927744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776489.879977:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664927744/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.879986:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776489.879989:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.879992:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664927744:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776489.879998:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664927872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776489.880019:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776489.880031:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776489/real 1775776489] req@ffff982974b2fb80 x1862032664927872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776489.880043:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664927872/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776489.880053:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776489.880056:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776489.880060:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664927872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776490.391148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776490.647259:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776491.415201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776491.671310:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776492.439136:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776492.695266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776493.463213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776493.719246:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776494.487143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776494.743253:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776494.999563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999576:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776494.999582:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999594:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999609:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999617:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776494.999664:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664928000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776494.999701:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999705:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776494.999707:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999712:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999715:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999719:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999730:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776494.999744:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664928128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776494.999744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664928000:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776494.999769:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999776:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776494.999777:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999781:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999784:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999788:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999792:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776494.999804:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664928256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776494.999814:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999817:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776494.999818:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999821:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999832:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999835:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999840:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776494.999850:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664928384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776494.999858:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999862:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776494.999865:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999869:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999872:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999875:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999880:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776494.999894:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664928512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776494.999895:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928128:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776494.999910:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999914:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776494.999917:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776494.999922:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999929:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999933:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999938:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776494.999943:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:2.0:1775776494.999951:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664928640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776494.999960:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776494.999965:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776494.999967:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776494.999970:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928384:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775776494.999975:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18262 00000100:00080000:2.0:1775776494.999979:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776494.999984:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776494.999990:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776494.999998:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:2.0:1775776495.000007:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664928768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776495.000023:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664928640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776495.000041:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664928768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776495.511227:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776495.767269:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776496.535232:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776496.791229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776497.559195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776497.815258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775776498.071407:0:404318:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:1.0:1775776498.071463:0:404318:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775776498.071485:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071543:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071554:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071563:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071572:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071581:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:1.0:1775776498.071590:0:404318:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776498.071694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b5c00 x1862032664928768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071728:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664928768/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071742:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071748:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664928768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776498.071762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b4700 x1862032664928512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664928512/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776498.071796:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b5f80 x1862032664928640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664928640/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071819:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071822:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664928640:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776498.071829:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b6d80 x1862032664928128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071839:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664928128/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071845:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071849:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928128:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776498.071858:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b4a80 x1862032664928256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664928256/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071875:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071878:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071881:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776498.071888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b5500 x1862032664928384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071897:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664928384/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071904:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071910:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664928384:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776498.071916:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776494/real 1775776498] req@ffff9829764b5180 x1862032664928000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776498.071927:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664928000/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776498.071933:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776498.071936:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776498.071939:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664928000:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776498.583211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776498.839204:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776499.607193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776499.863262:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776500.119184:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119192:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776500.119196:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776500.119205:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119218:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776500.119226:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776500.119264:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664928896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776500.119309:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119313:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776500.119315:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776500.119320:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119325:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776500.119329:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119339:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776500.119354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664929024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776500.119367:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776500.119370:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664928896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776500.119371:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776500.119375:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776500.119382:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119387:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776500.119392:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776500.119413:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664929152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776500.119424:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776500.119429:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776500.119434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119438:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776500.119442:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776500.119448:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119449:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776500.119464:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664929280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776500.119475:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119480:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776500.119484:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776500.119484:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b4e00 x1862032664928896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776500.119493:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119501:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776500.119506:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119512:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776500.119513:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664928896/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.119527:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776500.119530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664929408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776500.119534:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119543:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664928896:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776500.119552:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119557:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776500.119559:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776500.119564:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119569:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776500.119575:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929024:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776500.119576:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119586:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776500.119601:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664929536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776500.119610:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776500.119621:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776500.119625:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776500.119629:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776500.119629:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b7b80 x1862032664929024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776500.119638:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18267 00000100:00080000:2.0:1775776500.119647:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776500.119653:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664929024/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776500.119654:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776500.119663:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776500.119676:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776500.119678:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664929664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776500.119685:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929024:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776500.119705:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776500.119728:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776500.119739:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b7100 x1862032664929152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776500.119758:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664929152/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.119767:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776500.119770:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119773:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929152:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776500.119779:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929280:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776500.119798:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776500.119807:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b6a00 x1862032664929280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776500.119819:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664929280/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.119827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776500.119830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929280:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776500.119839:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776500.119857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776500.119866:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b4000 x1862032664929408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776500.119878:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664929408/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.119893:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776500.119896:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119900:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929408:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776500.119920:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664929536:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776500.119938:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776500.119948:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b6680 x1862032664929536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776500.119960:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664929536/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.119969:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776500.119972:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.119976:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664929536:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776500.119982:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664929664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776500.119998:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776500.120015:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776500/real 1775776500] req@ffff9829764b4e00 x1862032664929664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776500.120026:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664929664/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776500.120034:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776500.120038:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776500.120041:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664929664:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776500.631195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776500.887236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776501.655179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776501.911261:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776502.679209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776502.935350:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776503.703184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776503.959269:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776504.727167:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776504.983275:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776505.239232:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239240:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776505.239244:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239253:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239264:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239269:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776505.239304:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664929792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239330:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776505.239332:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239335:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239337:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239346:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776505.239355:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776505.239365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664929920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776505.239374:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664929792:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776505.239378:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776505.239381:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239385:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239387:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239389:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776505.239394:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776505.239402:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664930048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239409:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776505.239412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239415:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239418:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239420:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776505.239424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776505.239430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664930176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239439:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776505.239441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239443:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239446:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239451:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776505.239451:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775776505.239457:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776505.239465:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664930304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239471:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239473:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776505.239474:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239477:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00080000:1.0:1775776505.239480:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239483:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239487:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe3480 x1862032664929792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776505.239488:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776505.239499:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664930432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776505.239506:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776505.239509:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776505.239510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776505.239514:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18272 00000100:00100000:3.0:1775776505.239515:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664929792/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776505.239516:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776505.239521:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776505.239526:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776505.239538:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664930560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776505.239538:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239549:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239556:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664929792:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776505.239583:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929920:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776505.239602:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239611:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe1880 x1862032664929920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239619:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664929920/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239625:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239627:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239630:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664929920:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776505.239634:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776505.239644:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239650:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe2300 x1862032664930048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239658:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664930048/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239663:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239665:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239667:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776505.239671:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930176:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776505.239682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe2680 x1862032664930176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239695:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664930176/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930176:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776505.239707:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776505.239719:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe2a00 x1862032664930304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664930304/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239749:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239756:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776505.239763:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664930432:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776505.239783:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe0380 x1862032664930432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239803:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664930432/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239811:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239812:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664930432:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776505.239816:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664930560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776505.239829:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776505.239835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776505/real 1775776505] req@ffff982981fe0000 x1862032664930560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776505.239842:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664930560/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776505.239847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776505.239849:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776505.239851:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664930560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776505.751154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776506.007253:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776506.775181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776507.031273:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776507.799180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776508.055271:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776508.823187:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776509.079273:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776509.847162:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776510.103228:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776510.359187:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359195:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776510.359199:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359207:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359217:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359223:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776510.359258:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664930688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359280:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359284:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776510.359285:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359288:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359290:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359293:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776510.359300:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776510.359308:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664930816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359315:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359318:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776510.359319:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359322:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359323:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359329:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359331:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664930688:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775776510.359333:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776510.359343:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664930944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359349:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359351:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776510.359353:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359356:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359358:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359361:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776510.359365:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776510.359373:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664931072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359378:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776510.359383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359385:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359387:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359389:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776510.359392:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776510.359394:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776510.359402:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664931200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359409:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359411:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776510.359412:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776510.359415:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359418:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359421:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776510.359424:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776510.359428:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b5180 x1862032664930688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776510.359434:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664931328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776510.359451:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776510.359454:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776510.359455:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776510.359458:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664930688/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776510.359461:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18277 00000100:00080000:2.0:1775776510.359466:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776510.359471:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776510.359474:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776510.359480:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776510.359481:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664931456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776510.359488:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359496:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664930688:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776510.359503:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930816:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776510.359521:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359530:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b5500 x1862032664930816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359538:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664930816/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359543:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359548:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930816:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776510.359567:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776510.359577:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359584:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b4a80 x1862032664930944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664930944/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359597:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359599:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359601:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664930944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776510.359604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931072:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776510.359615:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359621:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b6d80 x1862032664931072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359628:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664931072/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359633:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359634:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359636:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931072:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776510.359639:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776510.359649:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359655:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b5f80 x1862032664931200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359661:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664931200/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359667:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359670:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776510.359687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664931328:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776510.359707:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359718:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b4700 x1862032664931328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359730:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664931328/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359739:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359741:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664931328:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776510.359748:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664931456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776510.359762:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776510.359769:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776510/real 1775776510] req@ffff9829764b5c00 x1862032664931456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776510.359776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664931456/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776510.359781:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776510.359783:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776510.359785:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664931456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776510.871179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776511.127279:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776511.895154:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776512.151225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776512.919178:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776513.175280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776513.943143:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776514.199263:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776514.967221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776515.223271:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776515.479204:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776515.479214:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776515.479219:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479230:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479243:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479249:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776515.479303:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664931584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479328:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776515.479335:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776515.479336:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479339:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479342:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479345:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776515.479365:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664931712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479373:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776515.479374:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664931584:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776515.479379:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776515.479382:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479385:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479387:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479389:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479392:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776515.479401:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664931840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479406:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776515.479408:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776515.479410:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479412:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479414:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479417:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479420:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776515.479426:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664931968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479430:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776515.479433:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776515.479434:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479437:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479439:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479442:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776515.479445:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479446:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776515.479455:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664932096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479460:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776515.479462:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776515.479463:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479466:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479468:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479470:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776515.479480:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664932224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776515.479485:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776515.479486:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b5c00 x1862032664931584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776515.479491:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776515.479494:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776515.479501:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18282 00000100:00080000:2.0:1775776515.479503:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776515.479505:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776515.479507:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776515.479516:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664932352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776515.479521:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664931584/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479530:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479534:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479537:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664931584:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776515.479560:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931712:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776515.479576:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479586:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b4700 x1862032664931712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664931712/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479605:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479608:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479612:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931712:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776515.479618:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776515.479637:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479646:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b5f80 x1862032664931840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479654:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664931840/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479664:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776515.479667:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931968:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776515.479680:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479685:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b6d80 x1862032664931968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479692:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664931968/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479697:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479698:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479700:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664931968:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776515.479703:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776515.479712:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479724:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b4a80 x1862032664932096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664932096/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479738:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479742:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776515.479744:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664932224:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776515.479754:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479759:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b5500 x1862032664932224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479769:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664932224/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479776:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479778:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479781:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664932224:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776515.479787:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664932352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776515.479806:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776515.479815:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776515/real 1775776515] req@ffff9829764b5180 x1862032664932352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776515.479826:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664932352/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776515.479833:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776515.479836:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776515.479839:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664932352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776515.991205:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776516.247205:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776517.015202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776517.271287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776518.039235:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776518.295384:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776519.063193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776519.319328:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776520.087221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776520.343340:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776520.599215:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599227:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776520.599232:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599256:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599270:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599278:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776520.599331:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664932480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776520.599360:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599363:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776520.599365:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599370:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599373:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599377:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776520.599388:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776520.599411:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664932480:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776520.599413:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664932608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776520.599427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599430:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776520.599432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599436:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599440:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599444:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776520.599448:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776520.599460:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664932736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776520.599468:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599483:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776520.599484:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599488:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599493:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776520.599494:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776520.599500:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776520.599506:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776520.599519:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664932864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776520.599539:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b4e00 x1862032664932480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776520.599540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599549:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776520.599553:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599561:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599565:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599569:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776520.599574:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776520.599584:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664932480/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.599595:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776520.599600:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664932992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776520.599601:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.599609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664932480:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776520.599618:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599622:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776520.599624:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599628:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599632:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599637:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776520.599641:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932608:192.168.201.118@tcp:38:kworker.0 00000008:00100000:2.0:1775776520.599644:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776520.599659:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664933120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776520.599669:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776520.599672:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776520.599674:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776520.599677:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18288 00000100:00080000:2.0:1775776520.599681:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776520.599685:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776520.599689:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776520.599699:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664933248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776520.599763:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.599777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b6680 x1862032664932608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.599788:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664932608/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.599795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.599798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.599801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932608:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776520.599806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776520.599826:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.599835:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b4000 x1862032664932736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.599846:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664932736/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.599853:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.599856:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.599858:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776520.599863:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932864:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776520.599973:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.599988:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b6a00 x1862032664932864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.600000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664932864/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.600008:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.600012:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.600015:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932864:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776520.600021:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776520.600040:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.600049:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b7100 x1862032664932992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.600098:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664932992/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.600108:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.600111:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.600114:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664932992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776520.600134:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664933120:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776520.600153:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.600163:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b7b80 x1862032664933120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.600174:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664933120/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.600182:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.600185:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.600188:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664933120:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776520.600193:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664933248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776520.600223:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776520.600233:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776520/real 1775776520] req@ffff9829764b4e00 x1862032664933248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776520.600244:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664933248/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776520.600257:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776520.600260:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776520.600263:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664933248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776521.111201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776521.367335:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776522.135200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776522.391255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776523.159220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776523.415273:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776524.183218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776524.439236:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776525.207238:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776525.463255:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776525.719237:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719251:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776525.719257:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719271:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719288:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719297:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776525.719351:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664933376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719379:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719382:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776525.719383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719387:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719390:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719393:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719403:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776525.719412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664933504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719418:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719421:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776525.719422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719425:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719426:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719429:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719434:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776525.719434:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664933376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:2.0:1775776525.719443:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664933632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719450:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719452:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776525.719453:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719456:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719459:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719462:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719465:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776525.719472:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664933760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719480:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719484:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776525.719486:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719491:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719518:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719523:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719528:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776525.719536:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:2.0:1775776525.719541:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664933888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719563:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719567:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776525.719568:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719576:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776525.719579:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719584:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776525.719593:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b5180 x1862032664933376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775776525.719598:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664934016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776525.719616:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776525.719619:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776525.719621:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776525.719624:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18293 00000100:00080000:2.0:1775776525.719628:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776525.719632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664933376/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776525.719634:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776525.719642:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775776525.719648:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.719654:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:2.0:1775776525.719657:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664934144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776525.719661:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664933376:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776525.719672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933504:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776525.719700:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.719715:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b5500 x1862032664933504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.719732:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664933504/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.719740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.719743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.719745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933504:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776525.719772:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933632:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776525.719792:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.719802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b4a80 x1862032664933632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.719812:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664933632/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.719843:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.719847:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.719849:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776525.719855:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933760:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776525.719875:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.719884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b6d80 x1862032664933760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.719895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664933760/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.719907:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.719910:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.719913:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933760:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776525.719918:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776525.719933:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.719942:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b5f80 x1862032664933888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.719951:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664933888/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.719958:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.719961:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.719964:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664933888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776525.719969:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934016:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776525.719983:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.719990:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b4700 x1862032664934016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.720000:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664934016/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.720006:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.720009:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.720013:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934016:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776525.720017:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776525.720032:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776525.720039:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776525/real 1775776525] req@ffff9829764b5c00 x1862032664934144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776525.720049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664934144/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776525.720056:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776525.720059:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776525.720062:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776526.231212:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776526.487295:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776527.255200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776527.511304:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776528.279169:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776528.535308:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776529.303176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776529.559473:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776530.327213:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776530.583298:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776530.839309:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839316:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776530.839320:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839328:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:3.0:1775776530.839337:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776530.839342:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776530.839375:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664934272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776530.839402:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839405:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776530.839406:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839418:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:3.0:1775776530.839422:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776530.839426:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776530.839435:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776530.839449:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664934400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776530.839461:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664934272:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776530.839462:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776530.839472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839477:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:3.0:1775776530.839480:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776530.839484:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776530.839490:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776530.839503:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664934528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776530.839513:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839517:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776530.839518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839521:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:3.0:1775776530.839526:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775776530.839527:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776530.839532:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776530.839538:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776530.839554:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664934656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776530.839563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9f800 x1862032664934272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776530.839582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839586:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776530.839588:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776530.839591:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664934272/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776530.839595:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:3.0:1775776530.839604:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776530.839607:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776530.839610:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776530.839615:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000008:00100000:3.0:1775776530.839616:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776530.839620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664934272:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776530.839631:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664934784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776530.839641:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776530.839646:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934400:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776530.839647:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776530.839658:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839667:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18298 00000800:00000100:1.0:1775776530.839669:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776530.839673:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776530.839678:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776530.839683:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776530.839684:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9c700 x1862032664934400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776530.839694:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664934912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776530.839699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664934400/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776530.839706:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776530.839713:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776530.839715:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776530.839716:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776530.839722:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18298 00000100:00080000:1.0:1775776530.839726:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776530.839727:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776530.839746:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776530.839748:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934400:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775776530.839750:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776530.839758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776530.839762:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664935040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776530.839782:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776530.839794:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9c380 x1862032664934528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776530.839807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664934528/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776530.839815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776530.839818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776530.839821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776530.839826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934656:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776530.839841:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776530.839849:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9d500 x1862032664934656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776530.839859:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664934656/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776530.839866:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776530.839869:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776530.839872:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934656:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776530.839877:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776530.839892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776530.839900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9fb80 x1862032664934784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776530.839910:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664934784/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776530.839917:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776530.839920:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776530.839923:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664934784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776530.839937:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934912:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776530.839951:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776530.839959:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9d880 x1862032664934912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776530.839970:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664934912/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776530.839977:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776530.839980:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776530.839982:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664934912:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776530.839987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776530.840001:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776530.840008:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776530/real 1775776530] req@ffff982980f9dc00 x1862032664935040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776530.840025:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664935040/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776530.840033:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776530.840036:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776530.840038:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776531.351189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776531.607250:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776532.375180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776532.631321:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776533.399208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776533.655266:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776534.423180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:3.0:1775776534.679222:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776535.447198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776535.703221:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776535.959205:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959215:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776535.959219:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959239:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959250:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776535.959258:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776535.959295:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664935168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776535.959325:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959329:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776535.959331:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959335:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959341:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776535.959346:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776535.959356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776535.959362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664935168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776535.959372:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664935296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776535.959387:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959390:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776535.959392:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959395:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959399:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776535.959403:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776535.959407:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776535.959418:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664935424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776535.959428:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959431:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776535.959433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776535.959437:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776535.959441:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959445:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776535.959449:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776535.959455:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776535.959464:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664935552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776535.959475:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe0000 x1862032664935168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776535.959496:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959501:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776535.959503:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959509:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00100000:3.0:1775776535.959510:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664935168/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776535.959515:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776535.959522:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:3.0:1775776535.959528:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775776535.959529:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775776535.959536:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959541:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664935168:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776535.959544:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664935680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776535.959554:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959557:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776535.959559:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959562:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959566:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776535.959570:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935296:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776535.959574:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776535.959581:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776535.959593:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664935808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776535.959603:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776535.959604:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776535.959608:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776535.959610:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776535.959614:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18303 00000100:00080000:1.0:1775776535.959618:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776535.959619:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe0380 x1862032664935296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776535.959626:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776535.959634:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776535.959650:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664935296/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776535.959660:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664935936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776535.959682:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959693:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935296:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776535.959700:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935424:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776535.959722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776535.959732:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe2a00 x1862032664935424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776535.959743:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664935424/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776535.959750:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959753:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959757:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776535.959762:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935552:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776535.959781:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776535.959790:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe2680 x1862032664935552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776535.959800:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664935552/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776535.959808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959811:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959814:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935552:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776535.959819:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935680:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776535.959837:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776535.959846:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe2300 x1862032664935680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776535.959853:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664935680/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776535.959857:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959859:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664935680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776535.959873:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935808:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776535.959883:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776535.959889:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe1880 x1862032664935808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776535.959895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664935808/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776535.959899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959901:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959903:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935808:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776535.959906:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935936:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776535.959914:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776535.959919:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776535/real 1775776535] req@ffff982981fe3480 x1862032664935936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776535.959925:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664935936/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776535.959929:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776535.959931:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776535.959932:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664935936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776536.471180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776536.727296:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776537.495219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776537.751271:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776538.519216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776538.775292:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776539.543220:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776539.799240:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776540.567199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776540.823270:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776541.079233:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079245:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776541.079250:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079263:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:1.0:1775776541.079278:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776541.079287:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776541.079338:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664936064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776541.079368:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079372:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776541.079374:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079378:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:1.0:1775776541.079383:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776541.079388:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776541.079399:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776541.079409:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664936064:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776541.079414:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664936192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776541.079426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079430:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776541.079432:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079436:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:1.0:1775776541.079440:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776541.079444:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776541.079448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776541.079459:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664936320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776541.079468:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079471:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776541.079472:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775776541.079477:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776541.079480:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:1.0:1775776541.079484:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776541.079488:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776541.079494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776541.079505:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664936448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776541.079514:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe3b80 x1862032664936064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776541.079516:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079531:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775776541.079531:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664936064/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776541.079535:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079544:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:3.0:1775776541.079546:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776541.079549:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776541.079564:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079568:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664936064:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776541.079572:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776541.079578:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776541.079590:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664936576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776541.079594:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936192:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776541.079609:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776541.079612:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776541.079614:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079620:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18308 00000800:00000100:3.0:1775776541.079620:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776541.079628:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776541.079632:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079635:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe1f80 x1862032664936192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776541.079638:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776541.079648:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664936192/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776541.079653:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664936704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776541.079657:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079664:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079670:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936192:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776541.079672:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776541.079676:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936320:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776541.079678:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776541.079680:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776541.079685:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18308 00000100:00080000:1.0:1775776541.079690:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775776541.079692:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776541.079697:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079701:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe0a80 x1862032664936320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775776541.079703:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776541.079714:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664936320/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776541.079720:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664936832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776541.079725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079733:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079740:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776541.079753:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936448:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776541.079774:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079783:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe0e00 x1862032664936448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776541.079792:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664936448/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776541.079798:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079801:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079803:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936448:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776541.079814:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776541.079826:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079831:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe1c00 x1862032664936576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776541.079837:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664936576/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776541.079842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664936576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776541.079849:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664936704:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776541.079857:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079861:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe1500 x1862032664936704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776541.079868:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664936704/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776541.079872:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079876:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664936704:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776541.079879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664936832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776541.079889:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776541.079903:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776541/real 1775776541] req@ffff982981fe1180 x1862032664936832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776541.079914:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664936832/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776541.079922:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776541.079925:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776541.079928:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664936832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776541.591202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776541.847244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776542.615202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776542.871255:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776543.639194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776543.895228:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776544.663173:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776544.919218:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776545.687159:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:1.0:1775776545.943223:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776546.199195:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199202:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776546.199206:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199214:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:3.0:1775776546.199224:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199229:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776546.199263:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664936960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199292:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199296:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776546.199298:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199301:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:3.0:1775776546.199304:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199308:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776546.199320:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776546.199329:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664936960:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776546.199335:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664937088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199347:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199350:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776546.199352:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199355:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:3.0:1775776546.199359:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199363:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776546.199367:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776546.199376:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664937216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776546.199386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199391:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18313 00000800:00000100:1.0:1775776546.199393:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776546.199395:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199399:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776546.199403:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776546.199410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664937344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199418:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776546.199419:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199422:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:3.0:1775776546.199424:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199428:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199429:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9ea00 x1862032664936960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775776546.199431:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776546.199440:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664937472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199447:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776546.199450:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776546.199451:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776546.199454:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664936960/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199477:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776546.199482:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:1.0:1775776546.199483:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776546.199485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776546.199489:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664936960:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776546.199491:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776546.199499:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776546.199508:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664937600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776546.199516:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776546.199519:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937088:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776546.199520:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776546.199523:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776546.199529:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18313 00000100:00080000:3.0:1775776546.199531:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776546.199533:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776546.199536:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776546.199543:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664937728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776546.199550:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199563:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9e680 x1862032664937088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199576:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664937088/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199587:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199591:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937088:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776546.199597:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776546.199611:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199619:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9ed80 x1862032664937216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199630:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664937216/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199638:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199641:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199644:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776546.199649:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937344:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776546.199663:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199670:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9df80 x1862032664937344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199680:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664937344/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199691:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199694:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937344:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776546.199699:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776546.199711:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199719:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9f480 x1862032664937472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199729:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664937472/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199737:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199740:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199743:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776546.199756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664937600:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776546.199769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199777:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9d180 x1862032664937600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199787:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664937600/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199795:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199798:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199801:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664937600:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776546.199806:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664937728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776546.199818:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776546.199825:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776546/real 1775776546] req@ffff982980f9c000 x1862032664937728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776546.199835:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664937728/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776546.199842:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776546.199846:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776546.199848:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664937728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776546.711221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776546.967248:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776547.735192:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776547.991294:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776548.759200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776549.015239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776549.783193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776550.039272:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776550.807161:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776551.063243:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776551.319217:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319234:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776551.319239:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776551.319251:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319266:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319274:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776551.319321:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5c00 x1862032664937856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776551.319353:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319357:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776551.319359:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776551.319364:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319368:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319372:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776551.319396:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776551.319408:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4700 x1862032664937984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776551.319417:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319420:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776551.319422:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776551.319427:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319431:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319439:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319440:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664937856:192.168.201.118@tcp:250:kworker.0 00000008:00100000:2.0:1775776551.319443:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776551.319456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5f80 x1862032664938112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776551.319465:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776551.319470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776551.319473:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319476:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319480:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776551.319486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776551.319496:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6d80 x1862032664938240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776551.319504:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319508:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776551.319510:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776551.319514:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18318 00000800:00000100:1.0:1775776551.319519:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776551.319520:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319526:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776551.319531:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776551.319542:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4a80 x1862032664938368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775776551.319551:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319554:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776551.319556:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776551.319561:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b5c00 x1862032664937856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775776551.319564:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319570:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319575:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776551.319580:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776551.319583:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5c00 x1862032664937856/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319592:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775776551.319594:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5500 x1862032664938496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776551.319597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319604:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664937856:192.168.201.118@tcp:250:kworker.0 00000100:00080000:2.0:1775776551.319611:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776551.319614:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776551.319618:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776551.319624:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937984:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776551.319625:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18318 00000100:00080000:2.0:1775776551.319631:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776551.319635:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776551.319641:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776551.319652:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b5180 x1862032664938624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776551.319657:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319693:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b4700 x1862032664937984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319706:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4700 x1862032664937984/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319714:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319718:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319721:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664937984:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776551.319749:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938112:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776551.319769:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319780:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b5f80 x1862032664938112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319791:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5f80 x1862032664938112/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319799:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319803:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319806:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776551.319812:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938240:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776551.319830:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319838:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b6d80 x1862032664938240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319845:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6d80 x1862032664938240/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319851:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319853:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319863:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938240:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776551.319869:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776551.319887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319896:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b4a80 x1862032664938368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4a80 x1862032664938368/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319915:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319919:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319922:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776551.319927:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664938496:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776551.319943:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319950:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b5500 x1862032664938496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319957:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5500 x1862032664938496/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319961:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319963:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319965:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664938496:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776551.319968:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664938624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776551.319977:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776551.319981:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776551/real 1775776551] req@ffff9829764b5180 x1862032664938624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776551.319991:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b5180 x1862032664938624/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776551.319995:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776551.319997:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776551.319998:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664938624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776551.831230:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776552.087270:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776552.855182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776553.111201:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776553.879204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776554.135277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776554.903204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776555.159457:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776555.927194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776556.183254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776556.439222:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439243:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776556.439252:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439264:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439279:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439301:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776556.439348:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664938752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439381:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776556.439383:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439393:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439398:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439416:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776556.439432:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664938752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776556.439432:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664938880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439445:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776556.439449:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439452:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439455:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439457:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439461:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776556.439469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664939008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439475:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439477:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776556.439478:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439481:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439482:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439485:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439487:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776556.439495:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664939136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439502:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439504:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776556.439506:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439508:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439510:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439512:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439514:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775776556.439521:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776556.439522:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664939264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439529:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439531:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776556.439533:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439535:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439537:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439539:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439542:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776556.439548:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664939392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776556.439552:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776556.439554:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776556.439555:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776556.439558:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18323 00000100:00080000:3.0:1775776556.439561:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776556.439563:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776556.439567:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776556.439571:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9e300 x1862032664938752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776556.439576:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664939520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776556.439608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664938752/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439620:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439624:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439629:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664938752:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776556.439654:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938880:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776556.439676:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.439687:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9ce00 x1862032664938880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.439699:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664938880/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439717:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439720:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664938880:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776556.439725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939008:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776556.439743:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.439752:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9f100 x1862032664939008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.439761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664939008/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439768:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439771:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439774:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939008:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776556.439780:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939136:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776556.439797:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.439805:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9c000 x1862032664939136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.439814:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664939136/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439820:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439884:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439888:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939136:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776556.439894:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939264:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776556.439913:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.439922:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9d180 x1862032664939264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.439931:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664939264/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439938:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439941:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439943:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939264:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776556.439948:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664939392:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776556.439964:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.439972:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9f480 x1862032664939392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.439983:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664939392/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.439990:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.439993:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.439995:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664939392:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776556.440000:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664939520:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776556.440017:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776556.440027:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776556/real 1775776556] req@ffff982980f9df80 x1862032664939520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776556.440038:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664939520/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776556.440047:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776556.440054:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776556.440058:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664939520:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776556.951244:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776557.207268:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776557.975198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776558.231303:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776558.999193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776559.255276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776560.023202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776560.279277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776561.047177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776561.303373:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776561.559150:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559161:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776561.559166:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559190:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559232:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559240:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776561.559284:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664939648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559315:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559319:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776561.559321:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559325:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559329:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559333:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776561.559343:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776561.559362:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664939648:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776561.559363:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664939776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559378:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776561.559390:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559394:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559398:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559401:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776561.559406:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776561.559418:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664939904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559427:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776561.559433:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559437:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559440:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559443:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776561.559447:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776561.559456:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664940032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559464:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559468:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776561.559470:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559475:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559479:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559483:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776561.559489:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939776:192.168.201.118@tcp:38:kworker.0 00000008:00100000:3.0:1775776561.559489:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776561.559501:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664940160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559512:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559516:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776561.559518:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776561.559522:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559526:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776561.559532:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939904:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776561.559532:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776561.559539:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776561.559551:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664940288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776561.559561:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776561.559565:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776561.559567:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775776561.559571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940032:192.168.201.118@tcp:8:kworker.0 00000100:00080000:3.0:1775776561.559572:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18329 00000100:00080000:3.0:1775776561.559577:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776561.559580:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776561.559585:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776561.559604:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664940416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776561.559608:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776561.559641:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664940288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776561.559663:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664940416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776562.071218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776562.327296:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776563.095263:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776563.351299:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776564.119186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776564.375278:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775776564.631309:0:404319:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:0.0:1775776564.631345:0:404319:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775776564.631359:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631389:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631394:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631399:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631403:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631411:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:0.0:1775776564.631420:0:404319:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:1.0:1775776564.631440:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9d500 x1862032664940416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631465:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664940416/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631483:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631488:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631493:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664940416:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776564.631500:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9d880 x1862032664940160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631512:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664940160/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631520:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631524:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631527:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940160:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776564.631533:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9fb80 x1862032664940288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631543:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664940288/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631549:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631552:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631555:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664940288:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776564.631560:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9e680 x1862032664939776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664939776/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631581:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631583:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939776:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776564.631588:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9ea00 x1862032664939904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631597:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664939904/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631604:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631606:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631609:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664939904:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776564.631614:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9dc00 x1862032664940032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631623:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664940032/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631629:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631632:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631635:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940032:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776564.631641:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776561/real 1775776564] req@ffff982980f9ed80 x1862032664939648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776564.631652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664939648/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776564.631659:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776564.631662:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776564.631665:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664939648:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776565.143214:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776565.399299:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776566.167183:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776566.423282:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776566.679223:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679236:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776566.679241:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679254:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679271:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679280:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776566.679339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664940544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679376:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679381:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776566.679383:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679387:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679391:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679396:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679406:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776566.679420:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664940672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679429:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679432:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776566.679433:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679437:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679440:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679444:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679448:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776566.679457:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664940800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679465:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679468:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776566.679469:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679473:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679475:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679479:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679484:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776566.679493:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664940928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679502:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679506:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776566.679508:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679512:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679515:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679517:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679522:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776566.679530:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664941056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679539:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679543:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776566.679545:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679548:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679551:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679555:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679559:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776566.679568:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664941184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776566.679576:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776566.679579:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776566.679581:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776566.679585:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18334 00000100:00080000:3.0:1775776566.679588:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776566.679591:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776566.679595:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776566.679603:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664941312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776566.680128:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664940544:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775776566.680238:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680298:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9c380 x1862032664940544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680323:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664940544/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680336:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680341:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680346:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664940544:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776566.680356:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940672:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776566.680384:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680399:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9c700 x1862032664940672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680411:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664940672/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680420:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680424:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680428:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940672:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776566.680434:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940800:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776566.680454:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680472:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9f800 x1862032664940800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680484:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664940800/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680492:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680496:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680499:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940800:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776566.680505:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940928:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776566.680527:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680537:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9ed80 x1862032664940928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680548:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664940928/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680557:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680560:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680563:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664940928:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776566.680571:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941056:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776566.680592:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680604:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9dc00 x1862032664941056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680615:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664941056/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680623:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680626:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680629:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941056:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776566.680635:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664941184:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776566.680654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680664:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9ea00 x1862032664941184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680675:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664941184/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680683:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680686:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680689:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664941184:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776566.680696:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664941312:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776566.680716:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776566.680727:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776566/real 1775776566] req@ffff982980f9e680 x1862032664941312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776566.680745:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664941312/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776566.680753:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776566.680756:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776566.680759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664941312:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776567.191248:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776567.447248:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776568.215172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776568.471290:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776569.239226:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776569.495277:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776570.263180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776570.519232:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776571.287184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776571.543236:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776571.799158:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799164:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776571.799168:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799176:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:3.0:1775776571.799192:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776571.799198:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776571.799237:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664941440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776571.799262:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799265:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776571.799267:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799271:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:3.0:1775776571.799275:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776571.799279:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776571.799287:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776571.799302:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664941568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776571.799310:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664941440:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776571.799313:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799318:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776571.799320:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799324:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:3.0:1775776571.799328:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776571.799332:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776571.799337:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776571.799349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664941696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776571.799358:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799361:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776571.799363:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799367:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:3.0:1775776571.799371:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775776571.799373:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776571.799378:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776571.799398:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776571.799407:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9fb80 x1862032664941440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776571.799412:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664941824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776571.799432:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776571.799437:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664941440/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776571.799437:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776571.799441:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799450:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:1.0:1775776571.799454:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776571.799456:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776571.799461:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776571.799463:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799468:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664941440:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775776571.799470:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776571.799483:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664941952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776571.799495:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941568:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776571.799503:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799507:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776571.799509:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776571.799514:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18339 00000800:00000100:1.0:1775776571.799519:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776571.799519:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776571.799526:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776571.799532:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776571.799536:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9d880 x1862032664941568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776571.799546:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664942080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776571.799553:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664941568/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776571.799564:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776571.799569:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776571.799571:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776571.799571:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776571.799580:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00080000:3.0:1775776571.799580:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18339 00000100:00080000:3.0:1775776571.799587:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776571.799589:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941568:192.168.201.118@tcp:38:kworker.0 00000100:00080000:3.0:1775776571.799592:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776571.799599:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776571.799603:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776571.799608:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664942208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776571.799628:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9d500 x1862032664941696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776571.799652:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664941696/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776571.799660:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776571.799663:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776571.799666:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941696:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776571.799672:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941824:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776571.799687:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799695:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9df80 x1862032664941824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776571.799705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664941824/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776571.799720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776571.799723:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776571.799726:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941824:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776571.799739:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941952:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776571.799753:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799761:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9f480 x1862032664941952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776571.799771:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664941952/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776571.799778:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776571.799781:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776571.799784:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664941952:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776571.799789:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942080:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776571.799803:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799810:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9d180 x1862032664942080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776571.799821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664942080/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776571.799827:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776571.799830:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776571.799834:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942080:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776571.799838:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942208:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776571.799852:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776571.799859:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776571/real 1775776571] req@ffff982980f9c000 x1862032664942208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776571.799870:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664942208/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776571.799877:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776571.799880:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776571.799882:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942208:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776572.311189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776572.567547:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776573.335177:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776573.591225:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776574.359189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776574.615293:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776575.383287:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776575.639290:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776576.407199:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776576.663229:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776576.919162:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919169:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776576.919172:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919180:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919190:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919195:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776576.919225:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f100 x1862032664942336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776576.919246:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919249:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776576.919250:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919253:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919256:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919258:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919265:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776576.919272:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ce00 x1862032664942464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776576.919279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776576.919282:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919286:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919287:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919289:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919293:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776576.919304:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e300 x1862032664942592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776576.919305:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664942336:192.168.201.118@tcp:250:kworker.0 00000100:00080000:3.0:1775776576.919319:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919323:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776576.919324:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919328:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919332:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919336:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919341:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776576.919366:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c000 x1862032664942720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776576.919377:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919379:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776576.919380:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919385:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18344 00000800:00000100:1.0:1775776576.919388:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:3.0:1775776576.919389:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919394:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919397:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776576.919404:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d180 x1862032664942848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776576.919410:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776576.919413:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919415:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919418:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919420:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919423:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776576.919430:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f480 x1862032664942976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776576.919431:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9f100 x1862032664942336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776576.919442:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776576.919448:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776576.919451:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776576.919456:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18344 00000100:00080000:3.0:1775776576.919459:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776576.919461:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776576.919464:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776576.919469:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f100 x1862032664942336/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776576.919471:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9df80 x1862032664943104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776576.919500:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919506:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919511:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664942336:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776576.919519:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942464:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776576.919541:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919553:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9ce00 x1862032664942464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919565:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ce00 x1862032664942464/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919573:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919577:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919580:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942464:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776576.919603:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942592:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776576.919618:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919625:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9e300 x1862032664942592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919632:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e300 x1862032664942592/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919637:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919639:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919641:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942592:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776576.919644:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942720:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776576.919654:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919660:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9c000 x1862032664942720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c000 x1862032664942720/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919670:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919671:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919673:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942720:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776576.919676:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942848:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776576.919685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919690:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9d180 x1862032664942848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919696:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d180 x1862032664942848/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919700:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919702:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919703:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664942848:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776576.919706:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942976:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776576.919715:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919722:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9f480 x1862032664942976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919733:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f480 x1862032664942976/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919740:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919746:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664942976:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776576.919751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664943104:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776576.919772:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776576.919779:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776576/real 1775776576] req@ffff982980f9df80 x1862032664943104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776576.919786:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9df80 x1862032664943104/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776576.919791:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776576.919793:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776576.919795:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664943104:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776577.431366:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776577.687261:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776578.456350:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776578.711256:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776579.479172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776579.735226:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776580.503148:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776580.759288:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776581.527206:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776581.783445:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776582.039218:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039227:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776582.039232:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039241:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039252:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039258:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776582.039294:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664943232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776582.039324:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039328:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776582.039330:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039334:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039338:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039343:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039353:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776582.039366:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664943360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776582.039371:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664943232:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776582.039384:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039386:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776582.039387:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039392:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039395:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039397:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039401:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776582.039410:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664943488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776582.039415:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039417:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776582.039418:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039420:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039423:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039425:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039428:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776582.039435:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664943616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776582.039441:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776582.039444:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039448:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776582.039449:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039459:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039463:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039467:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039473:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776582.039479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2e680 x1862032664943232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776582.039486:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664943744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776582.039496:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664943232/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776582.039497:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039507:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776582.039510:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776582.039512:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776582.039516:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:1.0:1775776582.039519:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:0.0:1775776582.039522:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776582.039525:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664943232:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776582.039529:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039535:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776582.039546:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664943872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776582.039556:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943360:192.168.201.118@tcp:38:kworker.0 00000100:00080000:0.0:1775776582.039558:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776582.039564:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776582.039566:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776582.039571:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18349 00000100:00080000:0.0:1775776582.039574:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776582.039577:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776582.039581:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776582.039587:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776582.039591:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664944000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776582.039600:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2df80 x1862032664943360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039608:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664943360/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039617:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943360:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776582.039623:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943488:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776582.039634:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776582.039640:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2ca80 x1862032664943488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039646:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664943488/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039651:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039652:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943488:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776582.039658:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943616:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776582.039668:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776582.039676:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2fb80 x1862032664943616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039686:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664943616/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039693:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039696:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039699:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943616:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776582.039704:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943744:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776582.039722:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776582.039730:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2f480 x1862032664943744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039737:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664943744/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039741:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039743:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039745:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664943744:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776582.039756:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664943872:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776582.039766:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776582.039771:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2f100 x1862032664943872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664943872/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039784:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039789:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664943872:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776582.039795:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944000:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776582.039812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776582.039822:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776582/real 1775776582] req@ffff982974b2d180 x1862032664944000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776582.039833:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664944000/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776582.039855:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776582.039858:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776582.039861:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944000:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776582.551294:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776582.807334:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776583.575197:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776583.831280:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776584.599194:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776584.855244:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776585.623182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776585.879280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776586.647256:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776586.903291:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776587.159300:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776587.159314:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776587.159320:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159332:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159346:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159354:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776587.159404:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664944128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776587.159439:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776587.159443:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776587.159444:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159450:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159454:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159458:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776587.159469:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776587.159483:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664944256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776587.159493:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776587.159497:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775776587.159500:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664944128:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776587.159512:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159518:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159521:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159525:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776587.159529:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776587.159542:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664944384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776587.159557:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776587.159560:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776587.159562:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159566:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159569:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159573:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776587.159578:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776587.159588:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664944512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776587.159596:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775776587.159598:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776587.159603:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776587.159605:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159609:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159613:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159617:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776587.159622:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776587.159633:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664944640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776587.159642:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776587.159645:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe1180 x1862032664944128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776587.159646:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776587.159650:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159658:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159665:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159671:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776587.159676:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776587.159678:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664944128/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776587.159690:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664944768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776587.159692:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.159701:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776587.159706:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776587.159709:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664944128:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776587.159711:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776587.159715:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776587.159731:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18354 00000100:00080000:1.0:1775776587.159736:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776587.159741:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776587.159741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944256:192.168.201.118@tcp:38:kworker.0 00000008:00100000:1.0:1775776587.159762:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776587.159777:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664944896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775776587.159790:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.159806:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe1500 x1862032664944256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.159821:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664944256/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.159829:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.159832:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.159837:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944256:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776587.159842:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944384:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776587.159868:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.159879:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe1c00 x1862032664944384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.159891:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664944384/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.159899:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.159902:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.159905:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944384:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776587.159911:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944512:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776587.159945:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.159957:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe0e00 x1862032664944512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.159968:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664944512/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.159976:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.159978:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.159982:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944512:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776587.159987:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944640:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776587.160007:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.160018:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe0a80 x1862032664944640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.160030:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664944640/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.160038:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.160041:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.160044:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664944640:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776587.160049:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944768:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776587.160108:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.160119:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe1f80 x1862032664944768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.160131:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664944768/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.160139:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.160143:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.160146:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944768:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776587.160167:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944896:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776587.160187:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776587.160197:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776587/real 1775776587] req@ffff982981fe3b80 x1862032664944896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776587.160208:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664944896/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776587.160217:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776587.160220:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776587.160223:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664944896:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776587.671211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776587.928506:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776588.695195:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776588.951321:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776589.719222:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776589.980598:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776590.743168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776590.999509:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776591.767525:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776592.023281:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775776592.279315:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.279329:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775776592.279335:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.279349:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.279365:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.279374:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:2.0:1775776592.279423:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4e00 x1862032664945024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.279529:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664945024:192.168.201.118@tcp:250:kworker.0 00000800:00000100:3.0:1775776592.279607:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776592.279643:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.279647:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776592.279649:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.279654:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.279673:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.279678:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.279690:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776592.279702:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7b80 x1862032664945152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.279776:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b4e00 x1862032664945024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.279797:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4e00 x1862032664945024/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.279808:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.279813:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.279817:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664945024:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776592.279858:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945152:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776592.279892:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776592.279905:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b7b80 x1862032664945152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.279918:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7b80 x1862032664945152/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.279926:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.279929:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.279931:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945152:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776592.280305:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.280310:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776592.280313:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.280317:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.280322:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.280327:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.280333:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776592.280348:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b7100 x1862032664945280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.280397:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945280:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776592.280421:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776592.280443:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.280447:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775776592.280449:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.280453:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.280457:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.280462:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.280468:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776592.280482:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6a00 x1862032664945408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.280520:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b7100 x1862032664945280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.280534:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b7100 x1862032664945280/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.280542:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.280545:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.280548:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945280:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776592.280562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945408:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776592.280584:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776592.280595:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b6a00 x1862032664945408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.280606:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6a00 x1862032664945408/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.280614:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.280617:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.280620:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945408:192.168.201.118@tcp:8:kworker.0 00000100:00080000:2.0:1775776592.280711:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.280716:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775776592.280718:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.280722:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.280727:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.280731:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.280737:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775776592.280817:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b4000 x1862032664945536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.280882:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945536:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776592.280908:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:2.0:1775776592.280930:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.280934:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775776592.280936:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.280940:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.280945:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.280949:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.280962:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776592.280975:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664945664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.281015:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b4000 x1862032664945536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.281028:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b4000 x1862032664945536/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.281036:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.281039:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.281042:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664945536:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776592.281054:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664945664:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776592.281108:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776592.281121:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b6680 x1862032664945664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.281134:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664945664/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.281154:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.281158:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.281161:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664945664:192.168.201.118@tcp:38:kworker.0 00000100:00080000:2.0:1775776592.281389:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775776592.281394:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775776592.281396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775776592.281401:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18359 00000100:00080000:2.0:1775776592.281406:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775776592.281410:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:2.0:1775776592.281416:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775776592.281430:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9829764b6680 x1862032664945792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776592.281514:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664945792:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776592.281534:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776592.281547:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776592/real 1775776592] req@ffff9829764b6680 x1862032664945792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776592.281559:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9829764b6680 x1862032664945792/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776592.281568:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776592.281572:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776592.281575:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9829764b6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664945792:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776592.791201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776593.047331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776593.815704:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776594.071505:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776594.839306:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776595.095354:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776595.863202:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776596.119289:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776596.887200:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776597.143648:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776597.399776:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.399789:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776597.399795:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.399807:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.399822:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.399830:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776597.399875:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664945920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.399913:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.399916:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776597.399918:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.399923:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.399926:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.399931:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.399941:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776597.399954:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664946048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.399965:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.399968:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776597.399970:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.399973:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.399977:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.399980:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.399984:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776597.399994:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664946176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.400015:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.400019:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776597.400020:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.400024:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.400028:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.400031:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.400036:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776597.400046:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664946304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.400055:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.400059:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776597.400060:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.400090:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.400095:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.400099:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.400104:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776597.400114:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664946432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.400123:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.400126:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776597.400128:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.400131:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.400134:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.400138:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.400141:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776597.400149:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664946560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776597.400157:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776597.400160:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776597.400162:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776597.400165:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18364 00000100:00080000:1.0:1775776597.400169:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776597.400172:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776597.400177:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776597.400185:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664946688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776597.402528:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664945920:192.168.201.118@tcp:250:kworker.0 00000800:00000100:3.0:1775776597.402636:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.402677:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe3480 x1862032664945920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.402697:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664945920/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.402708:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.402713:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.402717:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664945920:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776597.402725:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946048:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776597.402747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.402766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe1880 x1862032664946048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.402778:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664946048/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.402785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.402789:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.402792:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946048:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776597.402799:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946176:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776597.402818:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.402827:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe2300 x1862032664946176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.402838:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664946176/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.402847:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.402850:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.402853:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946176:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776597.402859:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946304:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776597.402879:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.402888:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe2680 x1862032664946304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.402899:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664946304/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.402906:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.402909:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.402912:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946304:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776597.402918:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946432:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776597.402936:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.402947:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe2a00 x1862032664946432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.402958:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664946432/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.402966:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.402969:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.402972:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946432:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776597.402979:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664946560:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776597.402995:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.403004:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe0380 x1862032664946560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.403016:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664946560/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.403024:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.403028:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.403030:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664946560:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776597.403036:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664946688:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776597.403053:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776597.403062:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776597/real 1775776597] req@ffff982981fe0000 x1862032664946688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776597.403102:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664946688/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776597.403110:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776597.403113:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776597.403116:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664946688:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776597.911189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776598.167593:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776598.935201:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776599.191249:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776599.959978:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776600.215241:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776600.983181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776601.239290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776602.007219:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776602.263276:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776602.519191:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519200:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776602.519204:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519213:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519224:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519243:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776602.519284:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664946816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519318:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519322:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776602.519324:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519328:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519331:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519335:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519343:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776602.519354:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664946944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519362:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519365:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776602.519366:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519370:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519372:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519375:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519378:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776602.519385:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664947072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519392:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519395:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776602.519396:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519399:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519402:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519405:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519409:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776602.519415:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664947200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519423:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519427:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776602.519428:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519432:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519435:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519438:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519442:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776602.519450:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664946816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776602.519451:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664947328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519465:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519468:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776602.519470:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519473:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519476:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519495:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519501:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776602.519514:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664947456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776602.519523:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776602.519526:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776602.519528:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776602.519532:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18370 00000100:00080000:1.0:1775776602.519536:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776602.519540:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:3.0:1775776602.519544:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:1.0:1775776602.519547:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776602.519561:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664947584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776602.519584:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe0000 x1862032664946816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519603:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664946816/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519612:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519615:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664946816:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776602.519625:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946944:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776602.519646:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519657:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe0380 x1862032664946944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519667:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664946944/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519675:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519678:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519681:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664946944:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776602.519687:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947072:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776602.519704:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519713:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe2a00 x1862032664947072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519722:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664947072/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519730:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519732:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519735:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947072:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776602.519741:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947200:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776602.519758:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519766:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe2680 x1862032664947200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519777:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664947200/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519785:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519788:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519791:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947200:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776602.519815:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947328:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776602.519831:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519840:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe2300 x1862032664947328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519852:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664947328/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519860:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519863:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519866:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947328:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776602.519871:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664947456:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776602.519887:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519895:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe1880 x1862032664947456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519906:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664947456/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519913:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519916:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519919:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664947456:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776602.519924:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664947584:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776602.519939:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776602.519946:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776602/real 1775776602] req@ffff982981fe3480 x1862032664947584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776602.519957:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664947584/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776602.519965:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776602.519968:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776602.519970:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664947584:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776603.031147:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776603.287852:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776604.055242:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776604.311295:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776605.079158:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776605.335242:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776606.103172:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776606.359254:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776607.127185:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776607.383228:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776607.639194:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639201:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776607.639205:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639212:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639221:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639226:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776607.639257:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d500 x1862032664947712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639279:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639281:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776607.639282:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639285:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639288:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639290:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776607.639297:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776607.639303:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9d880 x1862032664947840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639310:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639312:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776607.639314:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639317:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639319:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639323:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639325:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664947712:192.168.201.118@tcp:250:kworker.0 00000008:00100000:3.0:1775776607.639326:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776607.639332:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9fb80 x1862032664947968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639338:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639340:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776607.639341:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639344:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639345:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639348:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776607.639351:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776607.639358:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664948096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639363:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639366:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776607.639367:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639369:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639371:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639373:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776607.639375:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776607.639381:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664948224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639387:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639389:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776607.639390:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639403:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639405:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639410:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000800:00000100:1.0:1775776607.639411:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000008:00100000:3.0:1775776607.639415:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776607.639423:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664948352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776607.639429:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776607.639431:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776607.639432:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776607.639435:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18375 00000100:00080000:3.0:1775776607.639440:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776607.639442:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776607.639444:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776607.639451:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664948480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776607.639458:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9d500 x1862032664947712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639475:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d500 x1862032664947712/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639485:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639489:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639494:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664947712:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776607.639508:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947840:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776607.639532:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639544:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9d880 x1862032664947840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639558:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9d880 x1862032664947840/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639566:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639576:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639582:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947840:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776607.639603:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947968:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776607.639622:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639633:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9fb80 x1862032664947968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639644:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9fb80 x1862032664947968/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639652:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639655:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639658:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664947968:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776607.639664:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948096:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776607.639685:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9e680 x1862032664948096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639707:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664948096/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639725:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639728:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639732:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948096:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776607.639737:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948224:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776607.639755:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639764:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9ea00 x1862032664948224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639776:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664948224/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639786:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948224:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776607.639793:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664948352:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776607.639810:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9dc00 x1862032664948352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664948352/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639841:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639844:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639847:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664948352:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776607.639853:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664948480:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776607.639874:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776607.639884:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776607/real 1775776607] req@ffff982980f9ed80 x1862032664948480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776607.639895:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664948480/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776607.639903:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776607.639907:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776607.639910:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664948480:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776608.151179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776608.407434:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776609.175170:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776609.431347:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776610.199155:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776610.455290:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776611.223191:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776611.479254:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776612.247184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776612.503245:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776612.759242:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776612.759253:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776612.759259:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759271:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759287:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759294:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776612.759349:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664948608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776612.759380:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776612.759383:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776612.759385:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759389:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759393:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759397:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759407:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776612.759414:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664948608:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776612.759420:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664948736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776612.759430:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776612.759433:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776612.759435:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759439:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759442:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759444:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759449:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776612.759459:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664948864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776612.759467:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776612.759470:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776612.759472:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759475:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759478:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759481:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759486:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775776612.759492:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776612.759498:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664948992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776612.759506:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776612.759510:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776612.759511:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759515:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759518:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759520:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759524:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776612.759532:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664949120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776612.759540:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775776612.759541:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe3b80 x1862032664948608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776612.759546:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776612.759550:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776612.759556:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759561:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759565:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759568:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776612.759574:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664948608/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776612.759578:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664949248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776612.759601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776612.759609:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776612.759611:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776612.759615:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776612.759617:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776612.759619:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664948608:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776612.759623:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18380 00000100:00080000:1.0:1775776612.759629:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776612.759632:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776612.759636:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776612.759644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664949376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776612.759655:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948736:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776612.759682:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.759696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe1f80 x1862032664948736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.759708:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664948736/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.759716:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.759719:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.759722:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948736:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776612.759728:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948864:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776612.759749:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.759758:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe0a80 x1862032664948864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.759775:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664948864/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.759783:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.759785:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.759788:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948864:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776612.759794:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948992:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776612.759812:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.759821:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe0e00 x1862032664948992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.759832:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664948992/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.759840:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.759843:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.759846:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664948992:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776612.759852:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949120:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776612.759870:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.759880:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe1c00 x1862032664949120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.759890:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664949120/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.759903:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.759906:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.759909:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949120:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776612.759914:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664949248:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776612.759931:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.759940:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe1500 x1862032664949248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.759950:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664949248/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.759957:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.759960:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.759963:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664949248:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776612.759980:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664949376:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776612.759997:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776612.760005:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776612/real 1775776612] req@ffff982981fe1180 x1862032664949376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776612.760014:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664949376/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776612.760021:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776612.760024:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776612.760027:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664949376:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776613.271651:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776613.527311:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776614.295208:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776614.551462:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776615.319204:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776615.575293:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776616.343198:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776616.599280:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776617.367218:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776617.623335:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776617.879238:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879250:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776617.879255:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879267:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:1.0:1775776617.879283:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776617.879291:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776617.879337:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1180 x1862032664949504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776617.879372:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879375:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776617.879377:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879382:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:1.0:1775776617.879385:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776617.879389:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776617.879409:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776617.879418:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664949504:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776617.879425:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1500 x1862032664949632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776617.879436:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879440:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776617.879442:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879446:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:1.0:1775776617.879449:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776617.879453:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776617.879458:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776617.879469:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1c00 x1862032664949760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776617.879478:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879481:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776617.879483:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879488:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18385 00000800:00000100:3.0:1775776617.879489:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:1.0:1775776617.879494:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776617.879498:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776617.879504:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776617.879514:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0e00 x1862032664949888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776617.879524:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879528:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776617.879530:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879535:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:1.0:1775776617.879540:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775776617.879542:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe1180 x1862032664949504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776617.879546:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776617.879555:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776617.879568:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0a80 x1862032664950016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776617.879570:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1180 x1862032664949504/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776617.879582:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879588:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776617.879591:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776617.879599:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:3.0:1775776617.879601:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776617.879602:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776617.879608:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00080000:1.0:1775776617.879609:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776617.879614:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664949504:192.168.201.118@tcp:250:kworker.0 00000008:00100000:1.0:1775776617.879617:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776617.879630:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1f80 x1862032664950144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776617.879639:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776617.879643:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776617.879645:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775776617.879647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949632:192.168.201.118@tcp:38:kworker.0 00000100:00080000:1.0:1775776617.879652:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18385 00000100:00080000:1.0:1775776617.879656:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776617.879661:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776617.879666:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775776617.879673:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776617.879680:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3b80 x1862032664950272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776617.879697:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe1500 x1862032664949632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.879713:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1500 x1862032664949632/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.879720:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.879722:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.879725:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949632:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776617.879731:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949760:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776617.879747:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776617.879754:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe1c00 x1862032664949760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.879761:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1c00 x1862032664949760/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.879766:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.879768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.879770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949760:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776617.879773:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949888:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776617.879784:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776617.879790:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe0e00 x1862032664949888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.879807:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0e00 x1862032664949888/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.879815:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.879818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.879821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664949888:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776617.879826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950016:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776617.879846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776617.879857:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe0a80 x1862032664950016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.879865:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0a80 x1862032664950016/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.879871:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.879873:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.879875:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950016:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776617.879879:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664950144:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776617.879897:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776617.879917:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe1f80 x1862032664950144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.879929:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1f80 x1862032664950144/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.879935:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.879938:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.879940:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664950144:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776617.879960:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664950272:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776617.879994:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776617.880003:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776617/real 1775776617] req@ffff982981fe3b80 x1862032664950272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776617.880010:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3b80 x1862032664950272/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776617.880016:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776617.880018:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776617.880020:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664950272:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776618.391189:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776618.647287:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776619.415221:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776619.671305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776620.442024:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776620.695331:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776621.463225:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776621.719284:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776622.487157:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776622.743305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776622.999219:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999232:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776622.999287:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999300:0:374346:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999315:0:374346:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999322:0:374346:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776622.999369:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664950400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999408:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999412:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776622.999414:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999419:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999423:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999428:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999439:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776622.999453:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664950528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999476:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999479:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776622.999482:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999486:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999490:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999494:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999499:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776622.999511:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664950656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999521:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999524:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776622.999526:0:374346:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999530:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999533:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999544:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999550:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776622.999561:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664950784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999570:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999573:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776622.999574:0:374346:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999578:0:374346:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999582:0:374346:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999585:0:374346:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999589:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776622.999599:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664950912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999613:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999616:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776622.999618:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999621:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999624:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999628:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999634:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776622.999644:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664951040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776622.999653:0:374346:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776622.999656:0:374346:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776622.999658:0:374346:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776622.999662:0:374346:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18390 00000100:00080000:1.0:1775776622.999665:0:374346:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776622.999667:0:374346:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776622.999670:0:374346:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776622.999678:0:374346:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664951168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776622.999783:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664950400:192.168.201.118@tcp:250:kworker.0 00000800:00000100:3.0:1775776622.999856:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776622.999900:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776622/real 1775776622] req@ffff982981fe3480 x1862032664950400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776622.999919:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664950400/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776622.999940:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776622.999944:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776622.999947:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664950400:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776622.999952:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950528:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776622.999976:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776622.999985:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776622/real 1775776622] req@ffff982981fe1880 x1862032664950528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776622.999992:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664950528/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776622.999997:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000000:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000003:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950528:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776623.000009:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950656:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776623.000027:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776623.000037:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776622/real 1775776623] req@ffff982981fe2300 x1862032664950656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776623.000049:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664950656/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776623.000058:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000061:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000063:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950656:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776623.000087:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950784:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776623.000107:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776623.000119:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776623/real 1775776623] req@ffff982981fe2680 x1862032664950784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776623.000141:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664950784/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776623.000150:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000153:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000156:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950784:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776623.000161:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950912:192.168.201.118@tcp:8:kworker.0 00000800:00000100:3.0:1775776623.000181:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776623.000190:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776623/real 1775776623] req@ffff982981fe2a00 x1862032664950912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776623.000200:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664950912/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776623.000208:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000212:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000215:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664950912:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776623.000220:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951040:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776623.000246:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776623.000256:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776623/real 1775776623] req@ffff982981fe0380 x1862032664951040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776623.000267:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664951040/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776623.000275:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000279:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000282:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951040:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776623.000288:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951168:192.168.201.118@tcp:38:kworker.0 00000800:00000100:3.0:1775776623.000308:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:3.0:1775776623.000318:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776623/real 1775776623] req@ffff982981fe0000 x1862032664951168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776623.000336:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664951168/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776623.000345:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776623.000348:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776623.000351:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951168:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776623.511236:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776623.768181:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776624.535193:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776624.791266:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776625.559277:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776625.815305:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775776626.583209:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776626.839230:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776627.607318:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776627.863260:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775776628.119181:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119192:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775776628.119196:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119206:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00080000:1.0:1775776628.119219:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119225:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776628.119266:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0000 x1862032664951296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119291:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119294:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776628.119295:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119298:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00080000:1.0:1775776628.119301:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119304:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776628.119312:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776628.119322:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe0380 x1862032664951424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119331:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119334:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776628.119336:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119342:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00100000:3.0:1775776628.119343:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664951296:192.168.201.118@tcp:250:kworker.0 00000100:00080000:1.0:1775776628.119346:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119353:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776628.119358:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776628.119372:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2a00 x1862032664951552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119382:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119385:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775776628.119388:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119391:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00080000:1.0:1775776628.119395:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119400:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776628.119406:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775776628.119417:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2680 x1862032664951680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119426:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119429:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775776628.119431:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119434:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00080000:1.0:1775776628.119438:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119441:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776628.119447:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776628.119450:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776628.119461:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe2300 x1862032664951808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119471:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119475:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775776628.119476:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119481:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00080000:1.0:1775776628.119485:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119488:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776628.119493:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951552:192.168.201.118@tcp:38:kworker.0 00000008:00100000:1.0:1775776628.119494:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776628.119509:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe1880 x1862032664951936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775776628.119517:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775776628.119520:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775776628.119521:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775776628.119526:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18395 00000100:00100000:3.0:1775776628.119527:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951680:192.168.201.118@tcp:8:kworker.0 00000100:00080000:1.0:1775776628.119530:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776628.119536:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:1.0:1775776628.119539:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776628.119549:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982981fe3480 x1862032664952064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775776628.119562:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776628.119586:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776628.119604:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776628.631186:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776628.887239:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776629.655176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776629.912001:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776630.679184:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776630.935257:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775776631.191445:0:404316:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.118:988 00000400:00000100:2.0:1775776631.191490:0:404316:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.201.118@tcp at host 192.168.201.118 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775776631.191508:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776631.191549:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776631.191558:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776631.191567:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776631.191576:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000800:00000100:2.0:1775776631.191587:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776631.191590:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe2a00 x1862032664951552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775776631.191599:0:404316:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.118@tcp 00000100:00100000:3.0:1775776631.191633:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2a00 x1862032664951552/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191646:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191650:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191653:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951552:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776631.191658:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe2680 x1862032664951680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191666:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2680 x1862032664951680/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191671:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191674:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191676:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951680:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776631.191679:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe2300 x1862032664951808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191685:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe2300 x1862032664951808/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191689:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191690:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191692:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951808:192.168.201.118@tcp:8:kworker.0 00000100:00100000:3.0:1775776631.191696:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe0000 x1862032664951296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191702:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0000 x1862032664951296/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191707:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191708:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191710:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664951296:192.168.201.118@tcp:250:kworker.0 00000100:00100000:3.0:1775776631.191714:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe0380 x1862032664951424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191719:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe0380 x1862032664951424/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191723:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191726:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664951424:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776631.191746:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe1880 x1862032664951936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191752:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe1880 x1862032664951936/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191756:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191757:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191759:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664951936:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776631.191762:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776628/real 1775776631] req@ffff982981fe3480 x1862032664952064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775776631.191768:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982981fe3480 x1862032664952064/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775776631.191773:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775776631.191775:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:3.0:1775776631.191777:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982981fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952064:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776631.703175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776631.959231:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776632.727182:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776632.983245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775776633.239391:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239403:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775776633.239408:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239418:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239430:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239436:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:3.0:1775776633.239475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9f800 x1862032664952192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239507:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239511:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776633.239513:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239517:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239521:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239525:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239534:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776633.239545:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c700 x1862032664952320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239554:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239558:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776633.239559:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239563:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239566:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239569:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239573:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776633.239581:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9c380 x1862032664952448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239588:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239592:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775776633.239593:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239596:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239599:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239602:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239607:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776633.239616:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ed80 x1862032664952576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239623:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239627:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775776633.239628:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239632:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239635:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239637:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239642:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775776633.239651:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9dc00 x1862032664952704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239661:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239664:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775776633.239665:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239668:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239671:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239675:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239679:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776633.239687:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9ea00 x1862032664952832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775776633.239696:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775776633.239699:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775776633.239700:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775776633.239703:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18400 00000100:00080000:3.0:1775776633.239706:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775776633.239709:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:3.0:1775776633.239714:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775776633.239722:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982980f9e680 x1862032664952960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776633.240393:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664952192:192.168.201.118@tcp:250:kworker.0 00000800:00000100:1.0:1775776633.240493:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.240543:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9f800 x1862032664952192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.240564:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9f800 x1862032664952192/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.240577:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.240582:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.240587:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664952192:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776633.240596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952320:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776633.240619:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.240631:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9c700 x1862032664952320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.240642:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c700 x1862032664952320/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.240649:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.240653:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.240655:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952320:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776633.242277:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952448:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776633.242324:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.242350:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9c380 x1862032664952448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.242363:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9c380 x1862032664952448/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.242371:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.242376:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.242379:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952448:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776633.242517:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952576:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776633.242554:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.242567:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9ed80 x1862032664952576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.242577:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ed80 x1862032664952576/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.242584:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.242587:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.242590:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952576:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776633.242596:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952704:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776633.242619:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.242637:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9dc00 x1862032664952704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.242647:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9dc00 x1862032664952704/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.242655:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.242657:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.242660:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664952704:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776633.242665:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952832:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776633.242684:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.242694:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9ea00 x1862032664952832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.242705:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9ea00 x1862032664952832/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.242713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.242716:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.242718:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952832:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776633.242723:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952960:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776633.242741:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776633.242749:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776633/real 1775776633] req@ffff982980f9e680 x1862032664952960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776633.242759:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982980f9e680 x1862032664952960/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776633.242765:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776633.242768:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776633.242770:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982980f9e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664952960:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776633.751275:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776634.007228:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776634.775168:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776635.031249:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776635.799181:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776636.055258:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776636.823312:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776637.079262:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776637.847175:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776638.103254:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776638.359199:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359207:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776638.359210:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359219:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359230:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359234:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776638.359265:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2c380 x1862032664953088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359288:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359290:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776638.359291:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359295:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359297:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359299:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359306:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776638.359313:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d500 x1862032664953216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359319:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359322:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776638.359323:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359325:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359328:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359330:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359332:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776638.359339:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e300 x1862032664953344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359345:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359348:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776638.359349:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359352:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359354:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775776638.359355:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664953088:192.168.201.118@tcp:250:kworker.0 00000100:00080000:0.0:1775776638.359357:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359362:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776638.359369:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664953472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359375:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359377:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776638.359379:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359380:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359382:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359384:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359386:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776638.359393:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664953600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359399:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359402:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776638.359403:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359406:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359411:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359416:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359421:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775776638.359427:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:0.0:1775776638.359432:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664953728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776638.359444:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776638.359447:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776638.359449:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776638.359453:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18405 00000100:00080000:0.0:1775776638.359457:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776638.359460:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776638.359465:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776638.359475:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664953856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775776638.359479:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2c380 x1862032664953088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359508:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2c380 x1862032664953088/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359519:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359524:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359529:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664953088:192.168.201.118@tcp:250:kworker.0 00000100:00100000:1.0:1775776638.359536:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953216:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776638.359560:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359573:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2d500 x1862032664953216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359586:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d500 x1862032664953216/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359594:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359597:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359600:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953216:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776638.359632:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953344:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776638.359653:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359665:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2e300 x1862032664953344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359677:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e300 x1862032664953344/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359685:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359688:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359691:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953344:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776638.359698:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953472:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776638.359718:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359728:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2d180 x1862032664953472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359740:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664953472/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359747:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359749:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359752:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953472:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776638.359758:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953600:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776638.359785:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359797:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2f100 x1862032664953600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359808:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664953600/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359816:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359818:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359821:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664953600:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776638.359826:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664953728:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776638.359846:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359881:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2f480 x1862032664953728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359894:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664953728/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359902:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359905:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359908:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664953728:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776638.359913:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664953856:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776638.359932:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776638.359942:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776638/real 1775776638] req@ffff982974b2fb80 x1862032664953856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776638.359954:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664953856/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776638.359961:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776638.359964:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776638.359967:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664953856:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776638.871190:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776639.127297:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776639.895446:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776640.151321:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776640.919216:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776641.175286:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776641.943265:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776642.199307:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776642.967211:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776643.223389:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775776643.479150:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2585f5f-e151-4a1a-b002-2b900558e303->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479162:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775776643.479168:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479182:0:227016:0:(import.c:509:import_select_connection()) MGC192.168.201.118@tcp: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479197:0:227016:0:(import.c:588:import_select_connection()) MGC192.168.201.118@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479206:0:227016:0:(import.c:623:import_select_connection()) MGC192.168.201.118@tcp: import ffff982942aaf800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:0.0:1775776643.479252:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2ca80 x1862032664953984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479286:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479290:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776643.479291:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479295:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479298:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479302:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff982942b6f800: import ffff982945b58000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776643.479311:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776643.479325:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2df80 x1862032664954112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479335:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479339:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776643.479341:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479345:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479348:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479352:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff982942b6f800: import ffff982945b5a000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776643.479356:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775776643.479370:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2e680 x1862032664954240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479380:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479384:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775776643.479386:0:227016:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479389:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479393:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479399:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff982942b6f800: import ffff982945b5e000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479407:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664953984:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776643.479408:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776643.479425:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2fb80 x1862032664954368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479436:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) 14dd209b-d605-4024-a2b7-27ec435c0914->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479440:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775776643.479441:0:227016:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479446:0:227016:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479450:0:227016:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479454:0:227016:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff982942b6f800: import ffff98294542a800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776643.479459:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681674 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775776643.479471:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f480 x1862032664954496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479481:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775776643.479484:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775776643.479485:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775776643.479493:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00080000:0.0:1775776643.479496:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479503:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775776643.479508:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9829493c7800: import ffff9829493c0000 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000008:00100000:0.0:1775776643.479514:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776643.479536:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2ca80 x1862032664953984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775776643.479541:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2f100 x1862032664954624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775776643.479582:0:227016:0:(pinger.c:229:ptlrpc_pinger_process_import()) df0ee634-793c-4832-831b-03749b568504->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775776643.479587:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2ca80 x1862032664953984/t0(0) o250->MGC192.168.201.118@tcp@192.168.201.118@tcp:26/25 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775776643.479589:0:227016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775776643.479592:0:227016:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775776643.479605:0:227016:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: connect to NID 192.168.201.118@tcp last attempt 18410 00000100:00080000:0.0:1775776643.479618:0:227016:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775776643.479621:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982942aaf800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775776643.479627:0:227016:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9829493c7800: import ffff9829493c5800 using connection 192.168.201.118@tcp/192.168.201.118@tcp 00000100:00080000:1.0:1775776643.479628:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479639:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2585f5f-e151-4a1a-b002-2b900558e303:404328:1862032664953984:192.168.201.118@tcp:250:kworker.0 00000008:00100000:0.0:1775776643.479644:0:227016:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681674 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775776643.479647:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:0.0:1775776643.479668:0:227016:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff982974b2d180 x1862032664954752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775776643.479678:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479691:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2df80 x1862032664954112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479704:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2df80 x1862032664954112/t0(0) o38->lustre-MDT0000-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479713:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b58000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479725:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479736:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954112:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776643.479751:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954240:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776643.479789:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479802:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2e680 x1862032664954240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479813:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2e680 x1862032664954240/t0(0) o38->lustre-MDT0001-mdc-ffff982942b6f800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479819:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5a000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479821:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479823:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954240:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776643.479845:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954368:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776643.479861:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479868:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2fb80 x1862032664954368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479875:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2fb80 x1862032664954368/t0(0) o8->lustre-OST0000-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479879:0:404328:0:(import.c:65:import_set_state_nolock()) ffff982945b5e000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479881:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479883:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954368:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776643.479887:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954496:192.168.201.118@tcp:8:kworker.0 00000800:00000100:1.0:1775776643.479896:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479902:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2f480 x1862032664954496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479908:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f480 x1862032664954496/t0(0) o8->lustre-OST0001-osc-ffff982942b6f800@192.168.201.118@tcp:28/4 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479912:0:404328:0:(import.c:65:import_set_state_nolock()) ffff98294542a800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479914:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479916:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:14dd209b-d605-4024-a2b7-27ec435c0914:404328:1862032664954496:192.168.201.118@tcp:8:kworker.0 00000100:00100000:1.0:1775776643.479919:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664954624:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776643.479927:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479932:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2f100 x1862032664954624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479938:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2f100 x1862032664954624/t0(0) o38->lustre-MDT0000-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479943:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479945:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479947:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664954624:192.168.201.118@tcp:38:kworker.0 00000100:00100000:1.0:1775776643.479950:0:404328:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664954752:192.168.201.118@tcp:38:kworker.0 00000800:00000100:1.0:1775776643.479959:0:404328:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.118@tcp 00000100:00100000:1.0:1775776643.479963:0:404328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775776643/real 1775776643] req@ffff982974b2d180 x1862032664954752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775776643.479968:0:404328:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff982974b2d180 x1862032664954752/t0(0) o38->lustre-MDT0001-mdc-ffff9829493c7800@192.168.201.118@tcp:12/10 lens 520/544 e 0 to 1 dl 1775776668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775776643.479972:0:404328:0:(import.c:65:import_set_state_nolock()) ffff9829493c5800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775776643.479974:0:404328:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.118@tcp failed (-110) 00000100:00100000:1.0:1775776643.479976:0:404328:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff982974b2d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:df0ee634-793c-4832-831b-03749b568504:404328:1862032664954752:192.168.201.118@tcp:38:kworker.0 00000100:00100000:3.0:1775776643.991210:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776644.247285:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776645.015179:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776645.271361:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776646.039240:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776646.295285:0:404832:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776647.063180:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776647.320847:0:404833:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775776648.087176:0:412860:0:(client.c:2691:ptlrpc_set_wait()) set ffff982965db1480 going to sleep for 0 seconds 00010000:00010000:0.0:1775776648.343245:0:407993:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 73171 lines, 73171 kept, 0 dropped, 0 bad.